fedmsg.config handles loading, processing and validation of all configuration.
The configuration values used at runtime are determined by checking in the following order
- Built-in defaults
- Config file (/etc/fedmsg-config.py)
- Command line arguments
For example, if a config value does not appear in either the config file or on the command line, then the built-in default is used. If a value appears in both the config file and as a command line argument, then the command line value is used.
You can print the runtime configuration to the terminal by using the fedmsg-config command implemented by fedmsg.commands.config.config().
Return the global argparse.ArgumentParser used by all fedmsg commands.
Extra arguments can be supplied with the declared_args argument.
Setup a runtime config dict by integrating the following sources (ordered by precedence):
- defaults
- config file
- command line arguments
If the fedmsg_command argument is False, no command line arguments are checked.
Initialize an instance of fedmsg.core.FedMsgContext.
The config is loaded with fedmsg.config.load_config() and updated by any keyword arguments. This config is used to initialize the context object.
The object is stored in a thread local as fedmsg.__local.__context.
int - An option to zeromq that specifies a hard limit on the maximum number of outstanding messages to be queued in memory before reaching an exceptional state.
For our pub/sub zeromq sockets, the exceptional state means dropping messages. See the upstream documentation for ZMQ_HWM and ZMQ_PUB.
A high_water_mark of 0 means “no limit” and is the recommended value for fedmsg. It is referenced when initializing sockets in fedmsg.init().
int - An option that specifies the size of a zeromq thread pool to handle I/O operations. See the upstream documentation for zmq_init.
This value is referenced when initializing the zeromq context in fedmsg.init().
float - A number of seconds to sleep after initializing and before sending any messages. Setting this to a value greater than zero is required so that zeromq doesn’t drop messages that we ask it to send before the pub socket is finished initializing.
Experimentation needs to be done to determine and sufficiently small and safe value for this number. 1 is definitely safe, but annoyingly large.
dict - A mapping of “service keys” to “zeromq endpoints”; the heart of fedmsg.
endpoints is “a list of possible addresses from which fedmsg can send messages.” Thus, “subscribing to the bus” means subscribing to every address listed in endpoints.
endpoints is also an index where a fedmsg process can look up what port it should bind to to begin emitting messages.
When fedmsg.init() is invoked, a “name” is determined. It is either passed explicitly, or guessed from the call stack. The name is combined with the hostname of the process and used as a lookup key in the endpoints dict.
When sending, fedmsg will attempt to bind to each of the addresses listed under its service key until it can succeed in acquiring the port. There needs to be as many endpoints listed as there will be processes * threads trying to publish messages for a given service key.
For example, the following config provides for four WSGI processes on bodhi on the machine app01 to send fedmsg messages.
>>> config = dict(
... endpoints={
... "bodhi.app01": [
... "tcp://app01.phx2.fedoraproject.org:3000",
... "tcp://app01.phx2.fedoraproject.org:3001",
... "tcp://app01.phx2.fedoraproject.org:3002",
... "tcp://app01.phx2.fedoraproject.org:3003",
... ],
... },
... )
If apache is configured to start up five WSGI processes, the fifth one will produce tracebacks complaining with IOError("Couldn't find an available endpoint.").
If apache is configured to start up four WSGI processes, but with two threads each, four of those threads will raise exceptions with the same complaints.
A process subscribing to the fedmsg bus will connect a zeromq SUB socket to every endpoint listed in the endpoints dict. Using the above config, it would connect to the four ports on app01.phx2.fedoraproject.org.
Note
This is possibly the most complicated and hardest to understand part of fedmsg. It is the black sheep of the design. All of the simplicity enjoyed by the python API is achieved at cost of offloading the complexity here.
Some work could be done to clarify the language used for “name” and “service key”. It is not always consistent in fedmsg.core.
list - A list of domain names for which to query SRV records to get the associated endpoints.
When using fedmsg.config.load_config(), the DNS lookup is done and the resulting endpoints are added to config[‘endpoint’][$DOMAINNAME]
For example, the following would query the endpoints for foo.example.com.
>>> config = dict(
... srv_endpoints=[foo.example.com]
...)
dict - A mapping of service keys, the same as for endpoints to replay endpoints, each key having only one. The replay endpoints are special ZMQ endpoints using a specific protocol to allow the client to request a playback of messages in case some have been dropped, for instance due to network failures.
If the service has a replay endpoint specified, fedmsg will automatically try to detect such failures and properly query the endpoint to get the playback if needed.
str - A list of special zeromq endpoints where the inbound, passive zmq SUB sockets for for instances of fedmsg-relay are listening.
Commands like fedmsg-logger actively connect here and publish their messages.
See Bus Topology and Commands for more information.
bool - If set to true, then fedmsg.core will try to sign every message sent using the machinery from fedmsg.crypto.
It is often useful to set this to False when developing. You may not have X509 certs or the tools to generate them just laying around. If disabled, you will likely want to also disable validate_signatures.
bool - If set to true, then the base class fedmsg.consumers.FedmsgConsumer will try to use fedmsg.crypto.validate() to validate messages before handing them off to the particular consumer for which the message is bound.
This is also used by fedmsg.meta to denote trustworthiness in the natural language representations produced by that module.
dict - This should be a mapping of certnames to cert prefixes.
The keys should be of the form <service>.<host>. For example: bodhi.app01.
The values should be the prefixes of cert/key pairs to be found in ssldir. For example, if bodhi-app01.stg.phx2.fedoraproject.org.crt and bodhi-app01.stg.phx2.fedoraproject.org.key are to be found in ssldir, then the value bodhi-app01.stg.phx2.fedoraproject.org should appear in the certnames dict.
Putting it all together, this value could be specified as follows:
certnames={
"bodhi.app01": "bodhi-app01.stg.phx2.fedoraproject.org",
# ... other certname mappings may follow here.
}
Note
This is one of the most cumbersome parts of fedmsg. The reason we have to enumerate all these redundant mappings between “service.hostname” and “service-fqdn” has to do with the limitations of reverse dns lookup. Case in point, try running the following on app01.stg inside Fedora Infrastructure’s environment.
>>> import socket
>>> print socket.getfqdn()
You might expect it to print “app01.stg.phx2.fedoraproject.org”, but it doesn’t. It prints “memcached04.phx2.fedoraproject.org”. Since we can’t rely on programatically extracting the fully qualified domain names of the host machine during runtime, we need to explicitly list all of the certs in the config.
dict - A dict mapping fully-qualified topic names to lists of cert names. If a message’s topic appears in the routing_policy and the name on its certificate does not appear in the associated list, then that message fails the validation process in fedmsg.crypto.
For example, a routing policy might look like this:
routing_policy={
"org.fedoraproject.prod.bodhi.buildroot_override.untag": [
"bodhi-app01.phx2.fedoraproject.org",
"bodhi-app02.phx2.fedoraproject.org",
"bodhi-app03.phx2.fedoraproject.org",
"bodhi-app04.phx2.fedoraproject.org",
],
}
The above loosely translates to “messages about bodhi buildroot overrides being untagged may only come from the first four app servers.” If a message with that topic bears a cert signed by any other name, then that message fails the validation process.
Expect that your routing_policy (if you define one) will become quite long. It defaults to the empty dict, {}.
list - A list of ircbot configuration dicts. This is the primary way of configuring the fedmsg-irc bot implemented in fedmsg.commands.ircbot.ircbot().
Each dict contains a number of possible options. Take the following example:
>>> config = dict(
... irc=[
... dict(
... network='irc.freenode.net',
... port=6667,
... nickname='fedmsg-dev',
... channel='fedora-fedmsg',
... timeout=120,
...
... make_pretty=True,
... make_terse=True,
...
... filters=dict(
... topic=['koji'],
... body=['ralph'],
... ),
... ),
... ],
... )
Here, one bot is configured. It is to connect to the freenode network on port 6667. The bot’s name will be fedmsg-dev and it will join the #fedora-fedmsg channel.
make_pretty specifies that colors should be used, if possible.
make_terse specifies that the “natural language” representations produced by fedmsg.meta should be echoed into the channel instead of raw or dumb representations.
The filters dict is not very smart. In the above case, any message that has ‘koji’ anywhere in the topic or ‘ralph’ anywhere in the JSON body will be discarded and not echoed into #fedora-fedmsg. This is an area that could use some improvement.
dict - A mapping of modname values to MIRC irc color names. For example:
>>> irc_color_lookup = {
... "fas": "light blue",
... "bodhi": "green",
... "git": "red",
... "tagger": "brown",
... "wiki": "purple",
... "logger": "orange",
... "pkgdb": "teal",
... "buildsys": "yellow",
... "planet": "light green",
... }
list - A list of twitter/statusnet configuration dicts. This is the primary way of configuring the fedmsg-tweet bot implemented in fedmsg.commands.tweet.tweet().
Each dict contains a number of possible options. Take the following example:
>>> tweet_endpoints=[
... tweet_settings=dict(
... base_url="http://api.twitter.com",
... consumer_key="123456789ABCDEF",
... consumer_secret="123456789ABCDEF",
... access_token_key="12345678ABCDEF",
... access_token_secret="1234567ABCDEF",
... ),
... dict(
... base_url="http://identi.ca/api",
... consumer_key="12345676ABCDEF",
... consumer_secret="12345678ABCDEF",
... access_token_key="12355ABCEEF",
... access_token_secret="123456ABCDEF",
... ),
... ],
The base_url entry specifies which service to use. The other options are all oauth credentials.
See https://dev.twitter.com/docs/auth/tokens-devtwittercom about getting credentials for twitter.com. You can get all four authn values from their site.
Statusnet is a bit more tricky. You’ll need to get your consumer_key and consumer_secret yourself from http://identi.ca/ and then perform the “oauth dance” with this python script in order to get your access_token_key and access_token_secret.
dict - A dictionary containing credentials to shorten links against http://bit.ly/. It must contain values for api_user and api_key which can be obtained from http://bit.ly/
This is used primarily for fedmsg.commands.tweet.tweet() but could in theory be used elsewhere (like in fedmsg.commands.ircbot.ircbot())
bool - When false, allow splats (‘*’) in topic names when subscribing. When true, disallow splats and accept only strict matches of topic names.
This is an argument to moksha and arose there to help abstract away differences between the “topics” of zeromq and the “routing_keys” of AMQP.