EM 13.x agent relocation

There is one crucial decision when you install new local agent for Enterprise Manager – the location of it. If you stick to the OFA recommendation and include the EM minor version in base path – you’re screwed 🙂 Because after the upgrade of EM (for example 13.2 -> 13.3) you won’t have a chance to change it and be left with obsolete one. So just don’t do it and you may save yourself a whole lot of pain (reading the rest of my rambling and/or worse following my footsteps 🙂 ) Strangely enough there is no straightforward way to change the oracle home location of the installed agent. There are 2 ways of doing that I’m aware of – one is inconvenient and possibly not usable and the other one not recommended by Oracle and really a road through hell. I’ll focus on the latter one of course 🙂
And I’ll skip the option to remove the agent altogether and reinstall, as it will force you to remove all dependent targets with their history – a thing we want to avoid.
But let’s start with the easy way:
First you need to install second agent on the host machine on a different port. EM won’t allow you to do it from GUI (host already has installed agent…), but it should be possible by CLI means. It’s possible to change the port of the existing agent (MOS Doc ID 1556548.1) but still you need to have open at least 2 ports on firewall – that is if you care about availability. Don’t promote or configure any targets on the new agent. Instead you need to relocate existing ones to it using emcli relocate_targets command. For full syntax see the documentation. Then decommission the old agent and you’re done – theoretically – I haven’t check it out, as I couldn’t afford the mentioned prerequisites.

Instead here’s what I did (don’t try it at home – work works better 😉 No, seriously test it first somewhere safe):
There is a procedure made by Oracle to relocate central agent base directory out of middleware (oms) home location, which was the default for EMCC <= 12cR2. It's documented in Appendix C of EMCC upgrade guide and involves running the following scripts:

/perl/bin/perl /sysman/install/create_plugin_list.pl -instancehome ...

and then

/perl/bin/perl /sysman/install/AgentMigrate.pl -instanceHome ... -newAgentBaseDir ...

Before moving on note the “Important” remark on the doc page:

Moving the agent base directory is recommended only for central agents (on all platforms, including Microsoft Windows), and not recommended for standalone Management Agents.

With that acknowledged let’s continue 🙂

My case was to move from following base

/u01/app/oracle/product/13.2.0/agent

to

/u01/app/oracle/product/agent

which means running the following commands for my gold image version named BP190430:

/u01/app/oracle/product/13.2.0/agent/GoldImage_BP190430/agent_13.2.0.0.0/perl/bin/perl /u01/app/oracle/product/13.2.0/agent/GoldImage_BP190430/agent_13.2.0.0.0/sysman/install/create_plugin_list.pl -instancehome /u01/app/oracle/product/13.2.0/agent/agent_inst

/u01/app/oracle/product/13.2.0/agent/GoldImage_BP190430/agent_13.2.0.0.0/perl/bin/perl /u01/app/oracle/product/13.2.0/agent/GoldImage_BP190430/agent_13.2.0.0.0/sysman/install/AgentMigrate.pl -instancehome /u01/app/oracle/product/13.2.0/agent/agent_inst -newAgentBaseDir /u01/app/oracle/product/agent

It seemed to work at first glance – agent status was up on console, paths updated, but upon closer inspection it turned out something was wrong – while the agent was restarted and running from new location emctl status agent returned the agent was … not running. The same message also showed up when trying to restart agent from EM. Killing agent processes and trying to start it myself resulted in error message suggesting the targets.xml was not found (from my memory). This hinted me to search agent script and configuration files for occurrences of old path and unfortunately it turned out they were still there in the following files:

agent_inst/bin/forwarderProxy
agent_inst/bin/emtgtctl
agent_inst/sysman/registry.xml
agent_inst/sysman/config/emd.properties

I dealt with it this way (only after I initially went through a cycle of fixing one thing and receiving a different error message – which I don’t remember now – each time on startup ):

cd /u01/app/oracle/product/agent/agent_inst
./bin/emctl stop agent
sed -i 's;product/13.2.0;product;g' bin/forwarderProxy bin/emtgtctl sysman/registry.xml sysman/config/emd.properties
rm sysman/log/*
./bin/emctl start agent

That made the agent start up properly. But it was only the beginning of troubles… When I tried to subscribe my agents for new golden image it turned out they were not on the list. This is caused by agents home not being discovered/added to EM. And of course I couldn’t do it for some reason (error I can’t remember now…) After some investigation I noticed the agent’s targets.xml file didn’t contain new agent home entry, despite EM showing it on agent home page.
The solution to this is to perform agent resynchronization. It should be possible to do it using emcli resyncAgent -agent="Agent Name", but I’m lucky and all it does for me is print “Error: null” message, so I had to do it manually…
First block corresponding agents in web console … and that didn’t work too, as there was supposedly some blackout already active – the one issued by AgentMigrate.pl script. For some reason it was not properly ended (despite logs showing otherwise). Quick look in agent_inst/sysman/emd/blackouts.xml confirmed it. Solution:

cd /u01/app/oracle/product/agent/agent_inst/bin
./emctl status blackout
echo -e "<Blackouts>\n</Blackouts>" > ../sysman/emd/blackouts.xml
./emctl clearstate agent
./emctl upload agent
./emctl reload agent
./emctl status blackout

So, again – block corresponding agents in web console and for each one do resynchronization (on agent home page menu Agent->Resynchronization) … possibly multiple times, as it often failed for the first time with errors like “Target operation is unavailable: Cleanup of previous target with the same name (oracle_database.DBNAME) is in progress”….
Then you still need to unblock agent despite checking ‘unblock on successful resynchronization’ option due to supposedly negligible error (EM13c: Agent Resync Job Completes With Error: Error deploying metric extensions: JBO-INVALID_TARGET: Invalid target /server/server_os:oracle_si_server_os (Doc ID 2183354.1)).

If there are any other issues like monitored target (DB) wrong status it often helps to clear state of agent:

cd /u01/app/oracle/product/agent/agent_inst/bin
./emctl stop agent
./emctl clearstate agent
./emctl start agent

And only after this … you can refresh the configuration of the new agent home to update patch level information 😀 Or just wait for it to update itself.

OK, finally it’s working properly. End of procedure. A bit painful, yes, but hey – it works 🙂 Feel free to test it (and consolidate/automate some of the steps I repeated in historical order).

P.S.
I’m really tired of WordPress “fixing” my ‘<’ and ‘>’ characters and then failing to display the content properly in code markup (cutting it out; I’m editing in raw HTML). If anyone got a solution for this I’d be obliged…