From edewata at redhat.com Sat Sep 1 17:04:32 2012 From: edewata at redhat.com (Endi Sukma Dewata) Date: Sat, 01 Sep 2012 12:04:32 -0500 Subject: [Freeipa-devel] Paging in Web UI In-Reply-To: <5040D018.2090902@redhat.com> References: <503CC7E1.4040004@redhat.com> <503CD2E2.6010005@redhat.com> <503CF09A.5030902@redhat.com> <503D0E9E.6000703@redhat.com> <503E1DD6.9050504@redhat.com> <503E486C.1030006@redhat.com> <5040D018.2090902@redhat.com> Message-ID: <50424020.4020304@redhat.com> On 8/31/2012 9:54 AM, Petr Vobornik wrote: >> How about this, when you open the search page for the first time it will >> do a regular search, no paging. Instead of showing the paging controls >> (Prev & Next buttons) we show "See more results..." If you click that it >> will rerun the search with paging (which will give us the total number >> of entries) and show the paging control. > > I think we are in an agreement. It's basically what I wrote in different > words. I just mentioned current size limit. Let me try to summarize. We are dealing with 3 different aspects: In the UI we will use 2 levels of search pages. In the initial page we will do a quick search to get a partial result. If you want to see the full result you have to click 'more'. The full result can displayed using multiple pages (like the current implementation) or single continuous page (the one I'm suggesting as an alternative solution). The multi-page solution will have paging controls (prev & next buttons, page links), the single-page won't. Between the UI and the server the we can use 2-step operation (UI does the paging, it gets the pkey list first then gets the visible entries, this is the current implementation) or 1-step operation (server does the paging, UI gets the entries one page at a time). Any of these options can be used to implement either multi-page or single-page solution above. In the server we can retrieve the LDAP data with a simple search (default size limit is 100, can be increased up to 2000), with SPR (unlimited size, see below), or with VLV (unlimited size, but requires predefined filter). Any of these options can be used with either 1-step or 2-step operations to get the pkey list or the entries. > User workflow when searching for record (the same thing reworded): > 1) open search page: VLV or normal search with empty filter > 1a) when VLV: user would see paging ui > 1b) when normal search: not sure if to show 'see more results' link > for switching to paging. For smaller setups (<2000) it might be there, > for larger rather not - pkey list would be truncated. For the initial page if we use the simple search (1b) the UI will look like this: http://edewata.fedorapeople.org/images/paging1.png Here the UI will get up to 100 entries (default size limit), then it displays them in a single page. There will be a scroll bar to scroll through these 100 entries, but it will not pull more entries. To see other entries you click 'more', it will go to full result (3) below. The simple search doesn't return the total number of entries, so we can't show the paging controls. If we use SPR or VLV in the initial page we will know the total number of entries, so we don't really need an initial page, we can use the same page as the full result (3) below. Not sure about pros & cons of using SPR/VLV in the initial page, probably they will consume more server resources compared to simple search. > 2) search for record: enter filter to textbox, normal search would be used This is the same as (1b) except that in (1b) the filter is empty. Similarly, if the entry you're looking for is not there, you can click the 'more' button to see the full result (3). Here we can also use SPR, but not VLV because VLV filter is fixed in the VLV configuration. I'm not sure if we can apply LDAP filter on top of VLV filter. > 3) record not listed: switch to paging by clicking on 'see more results' > Paging should be OK because most likely we would get less than 2000 > records. Otherwise filter should be improved. I guess we can't go around > this unless we increase the limit or use other mean of paging. According to this page we can use the nsslapd-pagedsizelimit to remove the global size limit on SPR: https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/User_Account_Management-Setting_Resource_Limits_Based_on_the_Bind_DN.html#Setting_Resource_Limits_Based_on_the_Bind_DN-Setting_Resource_Limits_Using_the_Command_Line So with SPR the server we can get the full result, one page at a time, no more 2000 entry limitation. If we use the multi-page solution it will look like this: http://edewata.fedorapeople.org/images/paging2.png Currently we use 20 entries per page. We can change that to 100, same as the default search limit, so the transition from the initial page to the full result will be seamless. Both SPR and VLV will return the total number of entries, so we can show the paging controls. We can use simple search too (like the current implementation) but it will be limited to 2000 entries. > 4) record still not listed: page to find the record using current paging > UI or some improved There's no need for (4) because we can get all entries with SPR or VLV. >>> Regarding the continuous paging: I would extend the page by clicking at >>> a bar at the end of the list and/or by mouse-scrolling at the end of the >>> list? >> >> Ideally the scroll bar size should correspond to the visible part of the >> entire page. So you should be able to go anywhere in the entire page by >> moving the scroll bar around (either by keyboard, mouse click, mouse >> drag, or mouse scroll). In this case no need to extend the page. >> The thing is that we might have to create a table with many empty rows, >> which could be big, and populate them if they become visible. Maybe >> there's a way to avoid creating empty rows, we'll need to investigate. > > Did you meant by 'continuous paging' this solution or the one when you > click on 'see more results' (or scroll to last 10% of a scrollbar) and > it would show you next page? I originally understood it as the latter > but now I'm not sure. There are several variations for the single-page solution, the first one will look like this: http://edewata.fedorapeople.org/images/paging3.png It looks like a single page, but the entries are pulled one page at a time. Suppose we have 5000 entries total, initially we'll show 100 entries. The rest are virtual entries which will be represented as empty rows (or one big row). If you scroll down it will calculate the new page number, then it will pull the entries for that page and populate the empty rows. To save memory, rows that are not visible can be made empty again. > This solution seems challenging to implement right. I'm also not sure > how usable it is because I don't remember if I ever used it somewhere. Yeah, it could be complicated to implement. > Anyway empty rows can be also replaced by rows with huge height. Right. But if we jump around to random pages we may have multiple 'big rows' that need to be maintained. >> Alternatively we could do like image search on bing.com. If you bring >> the scroll bar say within 10% from the bottom it will load the next >> page. The thing is it will load the page sequentially, and if you >> continue to the bottom you'll end up with a very big table. > > I like this with the Prev extension mentioned below. This is another variation of the single-page solution, it will look like this: http://edewata.fedorapeople.org/images/paging4.png The difference is we don't create 'big rows', but if you scroll near the bottom it will pull the next page and append the entries at the bottom and we keep the old entries at the top. The drawing shows 20 entries, but we can change it to 100. >> Another possibility, we can use the top 10% of the scroll bar as a Prev >> button, and the bottom %10 as the Next button, and we remove the pages >> that aren't visible anymore. But this is not much different than the >> current implementation, only different presentation. This is yet another variation of the single-page solution. It's similar to the previous one, but here we don't keep the old entries at the top. So if you go near the top it will pull the previous page and prepend the entries at the top. These last two options will be more limited than the current implementation because you can't jump to random pages, unless we make it such that it keeps scrolling as long as you keep the scroll bar near the top/bottom and only pull the page if you release it. We will also need a way to show where we are in the database. > I can image to use each of these 3 solutions for enabled paging. Also > switching between second and third solution might be configurable. > > Anyway. First I would like to make paging/not paging configurable and > switchable when needed. Then we can proceed to improving paging ui > (better pager, continuous paging and such). Yes, in other words it will be the switch between the initial page and the full result. -- Endi S. Dewata From alonbl at redhat.com Sat Sep 1 23:27:12 2012 From: alonbl at redhat.com (Alon Bar-Lev) Date: Sat, 1 Sep 2012 19:27:12 -0400 (EDT) Subject: [Freeipa-devel] java & startTLS session closure In-Reply-To: <59750573.4292255.1346538944864.JavaMail.root@redhat.com> Message-ID: <798838947.4292448.1346542032302.JavaMail.root@redhat.com> Hello, Not sure that it is IPA issue, however this happens to me when using IPA and not other LDAP servers. I am using the following program, which should work as far as java documentation outlines, and while trying to close the TLS session, the tls.close() function is not returning. Have someone bump into this? Using: fedora 17 repositories. freeipa-server-2.2.0-1.fc17.x86_64 Regards, Alon Bar-Lev --- LDAP1.java $ javac LDAP1.java && java LDAP1 BEFORE tls.close() --- import java.io.*; import java.net.*; import java.util.*; import java.security.cert.*; import javax.naming.*; import javax.naming.directory.*; import javax.naming.ldap.*; import javax.net.*; import javax.net.ssl.*; public class LDAP1 { public static SSLSocketFactory createUnsecureSocketFactory(String protocol) { // Install the all-trusting trust manager try { SSLContext sc = SSLContext.getInstance(protocol); sc.init( null, new TrustManager[]{ new X509TrustManager() { public java.security.cert.X509Certificate[] getAcceptedIssuers() { return new java.security.cert.X509Certificate[] {}; } public void checkClientTrusted( java.security.cert.X509Certificate[] certs, String authType) { } public void checkServerTrusted( java.security.cert.X509Certificate[] certs, String authType) { } } }, null ); return sc.getSocketFactory(); } catch(Exception e) { throw new RuntimeException(e); } } public static void main(String args[]) throws Exception { String url = args[0]; LdapContext ctx = null; StartTlsResponse tls = null; try { Hashtable env = new Hashtable(); env.put(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.ldap.LdapCtxFactory"); env.put(Context.PROVIDER_URL, url); ctx = new InitialLdapContext(env, null); tls = (StartTlsResponse)ctx.extendedOperation( new StartTlsRequest() ); SSLSession session = tls.negotiate( /* * DEBUG-BEGIN * this is required for debug only * disable certificate validation */ createUnsecureSocketFactory("TLSv1") /* DEBUG-END */ ); } finally { System.out.println("BEFORE tls.close()"); if (tls != null) { tls.close(); } System.out.println("AFTER tls.close()"); if (ctx != null) { ctx.close(); } } } } From sbose at redhat.com Sun Sep 2 17:04:22 2012 From: sbose at redhat.com (Sumit Bose) Date: Sun, 2 Sep 2012 19:04:22 +0200 Subject: [Freeipa-devel] [PATCHES] Remove dependencies to private samba libs Message-ID: <20120902170422.GC2071@localhost.localdomain> Hi, private samba libraries will become a new version with every new samba release. To avoid rebuilding of IPA whenever a new samba version is released the following series of patches removes the dependencies to private samba libraries by replacing the related calls with others. This should fix https://fedorahosted.org/freeipa/ticket/3013 bye, Sumit -------------- next part -------------- From 802cbf261c164371d6777ae0f6d7e7f2f305343f Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Fri, 24 Aug 2012 13:21:11 +0200 Subject: [PATCH 42/59] ipasam: cleanup explicit dependencies to samba libs --- daemons/ipa-sam/Makefile.am | 2 -- 1 Datei ge?ndert, 2 Zeilen entfernt(-) diff --git a/daemons/ipa-sam/Makefile.am b/daemons/ipa-sam/Makefile.am index ad7e516f0c94f82cc209ee55ff0b67c6a6bd54f9..ae0de4ecb7e4eb0298c87e6b544c643d7ba7f4ce 100644 --- a/daemons/ipa-sam/Makefile.am +++ b/daemons/ipa-sam/Makefile.am @@ -1,9 +1,7 @@ NULL = SAMBA40EXTRA_LIBS = $(SAMBA40EXTRA_LIBPATH) \ -lsmbldap \ - -lcliauth \ -lpdb \ - -lsamba-security \ -lsmbconf \ $(NULL) -- 1.7.11.4 -------------- next part -------------- From aaa717997f8869f9aa5308dfb97dee1f08b1e99a Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Fri, 24 Aug 2012 14:46:05 +0200 Subject: [PATCH 43/59] Make encode_ntlm_keys() public --- daemons/ipa-sam/Makefile.am | 3 +- .../ipa-slapi-plugins/ipa-pwd-extop/Makefile.am | 3 +- .../ipa-pwd-extop/ipapwd_encoding.c | 207 ++------------------ util/ipa_pwd.c | 2 + util/ipa_pwd.h | 11 ++ util/ipa_pwd_ntlm.c | 213 +++++++++++++++++++++ 6 Dateien ge?ndert, 242 Zeilen hinzugef?gt(+), 197 Zeilen entfernt(-) create mode 100644 util/ipa_pwd_ntlm.c diff --git a/daemons/ipa-sam/Makefile.am b/daemons/ipa-sam/Makefile.am index ae0de4ecb7e4eb0298c87e6b544c643d7ba7f4ce..11179276c17f9b0415fc5ad944b88bcbbb0529c3 100644 --- a/daemons/ipa-sam/Makefile.am +++ b/daemons/ipa-sam/Makefile.am @@ -6,7 +6,7 @@ SAMBA40EXTRA_LIBS = $(SAMBA40EXTRA_LIBPATH) \ $(NULL) KRB5_UTIL_DIR=../../util -KRB5_UTIL_SRCS=$(KRB5_UTIL_DIR)/ipa_krb5.c +KRB5_UTIL_SRCS=$(KRB5_UTIL_DIR)/ipa_krb5.c $(KRB5_UTIL_DIR)/ipa_pwd_ntlm.c INCLUDES = \ -I. \ @@ -48,6 +48,7 @@ ipasam_la_LIBADD = \ $(LDAP_LIBS) \ $(KRB5_LIBS) \ $(TALLOC_LIBS) \ + $(SSL_LIBS) \ $(SAMBAUTIL_LIBS) \ $(NDR_LIBS) \ $(SAMBA40EXTRA_LIBS) \ diff --git a/daemons/ipa-slapi-plugins/ipa-pwd-extop/Makefile.am b/daemons/ipa-slapi-plugins/ipa-pwd-extop/Makefile.am index bdc583566409cfd44c4f144469ca4605295f3286..f3bb589611efbd53eb2f0969ee858c2007829892 100644 --- a/daemons/ipa-slapi-plugins/ipa-pwd-extop/Makefile.am +++ b/daemons/ipa-slapi-plugins/ipa-pwd-extop/Makefile.am @@ -3,7 +3,8 @@ NULL = PLUGIN_COMMON_DIR=../common KRB5_UTIL_DIR= ../../../util KRB5_UTIL_SRCS = $(KRB5_UTIL_DIR)/ipa_krb5.c \ - $(KRB5_UTIL_DIR)/ipa_pwd.c + $(KRB5_UTIL_DIR)/ipa_pwd.c \ + $(KRB5_UTIL_DIR)/ipa_pwd_ntlm.c INCLUDES = \ -I. \ diff --git a/daemons/ipa-slapi-plugins/ipa-pwd-extop/ipapwd_encoding.c b/daemons/ipa-slapi-plugins/ipa-pwd-extop/ipapwd_encoding.c index 9c1623a3b28932fba48c878dc6084862a2ba7831..a92eaf0da33809e5a4b81c7f10a801e256880743 100644 --- a/daemons/ipa-slapi-plugins/ipa-pwd-extop/ipapwd_encoding.c +++ b/daemons/ipa-slapi-plugins/ipa-pwd-extop/ipapwd_encoding.c @@ -198,201 +198,6 @@ enc_error: return NULL; } - -#define KTF_DOS_CHARSET "CP850" /* same default as samba */ -#define KTF_UTF8 "UTF-8" -#define KTF_UCS2 "UCS-2LE" - -static const uint8_t parity_table[128] = { - 1, 2, 4, 7, 8, 11, 13, 14, 16, 19, 21, 22, 25, 26, 28, 31, - 32, 35, 37, 38, 41, 42, 44, 47, 49, 50, 52, 55, 56, 59, 61, 62, - 64, 67, 69, 70, 73, 74, 76, 79, 81, 82, 84, 87, 88, 91, 93, 94, - 97, 98,100,103,104,107,109,110,112,115,117,118,121,122,124,127, - 128,131,133,134,137,138,140,143,145,146,148,151,152,155,157,158, - 161,162,164,167,168,171,173,174,176,179,181,182,185,186,188,191, - 193,194,196,199,200,203,205,206,208,211,213,214,217,218,220,223, - 224,227,229,230,233,234,236,239,241,242,244,247,248,251,253,254 -}; - -static void lm_shuffle(uint8_t *out, uint8_t *in) -{ - out[0] = parity_table[in[0]>>1]; - out[1] = parity_table[((in[0]<<6)|(in[1]>>2)) & 0x7F]; - out[2] = parity_table[((in[1]<<5)|(in[2]>>3)) & 0x7F]; - out[3] = parity_table[((in[2]<<4)|(in[3]>>4)) & 0x7F]; - out[4] = parity_table[((in[3]<<3)|(in[4]>>5)) & 0x7F]; - out[5] = parity_table[((in[4]<<2)|(in[5]>>6)) & 0x7F]; - out[6] = parity_table[((in[5]<<1)|(in[6]>>7)) & 0x7F]; - out[7] = parity_table[in[6] & 0x7F]; -} - -struct ntlm_keys { - uint8_t lm[16]; - uint8_t nt[16]; -}; - -/* create the lm and nt hashes - newPassword: the clear text utf8 password - do_lm_hash: determine if LM hash is generated - do_nt_hash: determine if NT hash is generated - keys[out]: array with generated hashes -*/ -static int encode_ntlm_keys(char *newPasswd, - bool do_lm_hash, - bool do_nt_hash, - struct ntlm_keys *keys) -{ - int ret = 0; - - /* do lanman first */ - if (do_lm_hash) { - iconv_t cd; - size_t cs, il, ol; - char *inc, *outc; - char *upperPasswd; - char *asciiPasswd; - DES_key_schedule schedule; - DES_cblock deskey; - DES_cblock magic = "KGS!@#$%"; - - /* TODO: must store the dos charset somewhere in the directory */ - cd = iconv_open(KTF_DOS_CHARSET, KTF_UTF8); - if (cd == (iconv_t)(-1)) { - ret = -1; - goto done; - } - - /* the lanman password is upper case */ - upperPasswd = (char *)slapi_utf8StrToUpper((unsigned char *)newPasswd); - if (!upperPasswd) { - iconv_close(cd); - ret = -1; - goto done; - } - il = strlen(upperPasswd); - - /* an ascii string can only be smaller than or equal to an utf8 one */ - ol = il; - if (ol < 14) ol = 14; - asciiPasswd = calloc(ol+1, 1); - if (!asciiPasswd) { - slapi_ch_free_string(&upperPasswd); - iconv_close(cd); - ret = -1; - goto done; - } - - inc = upperPasswd; - outc = asciiPasswd; - cs = iconv(cd, &inc, &il, &outc, &ol); - if (cs == -1) { - ret = -1; - slapi_ch_free_string(&upperPasswd); - free(asciiPasswd); - iconv_close(cd); - goto done; - } - - /* done with these */ - slapi_ch_free_string(&upperPasswd); - iconv_close(cd); - - /* we are interested only in the first 14 ASCII chars for lanman */ - if (strlen(asciiPasswd) > 14) { - asciiPasswd[14] = '\0'; - } - - /* first half */ - lm_shuffle(deskey, (uint8_t *)asciiPasswd); - - DES_set_key_unchecked(&deskey, &schedule); - DES_ecb_encrypt(&magic, (DES_cblock *)keys->lm, - &schedule, DES_ENCRYPT); - - /* second half */ - lm_shuffle(deskey, (uint8_t *)&asciiPasswd[7]); - - DES_set_key_unchecked(&deskey, &schedule); - DES_ecb_encrypt(&magic, (DES_cblock *)&(keys->lm[8]), - &schedule, DES_ENCRYPT); - - /* done with it */ - free(asciiPasswd); - - } else { - memset(keys->lm, 0, 16); - } - - if (do_nt_hash) { - iconv_t cd; - size_t cs, il, ol, sl; - char *inc, *outc; - char *ucs2Passwd; - MD4_CTX md4ctx; - - /* TODO: must store the dos charset somewhere in the directory */ - cd = iconv_open(KTF_UCS2, KTF_UTF8); - if (cd == (iconv_t)(-1)) { - ret = -1; - goto done; - } - - il = strlen(newPasswd); - - /* an ucs2 string can be at most double than an utf8 one */ - sl = ol = (il+1)*2; - ucs2Passwd = calloc(ol, 1); - if (!ucs2Passwd) { - ret = -1; - iconv_close(cd); - goto done; - } - - inc = newPasswd; - outc = ucs2Passwd; - cs = iconv(cd, &inc, &il, &outc, &ol); - if (cs == -1) { - ret = -1; - free(ucs2Passwd); - iconv_close(cd); - goto done; - } - - /* done with it */ - iconv_close(cd); - - /* get the final ucs2 string length */ - sl -= ol; - - ret = MD4_Init(&md4ctx); - if (ret == 0) { - ret = -1; - free(ucs2Passwd); - goto done; - } - ret = MD4_Update(&md4ctx, ucs2Passwd, sl); - if (ret == 0) { - ret = -1; - free(ucs2Passwd); - goto done; - } - ret = MD4_Final(keys->nt, &md4ctx); - if (ret == 0) { - ret = -1; - free(ucs2Passwd); - goto done; - } - - } else { - memset(keys->nt, 0, 16); - } - - ret = 0; - -done: - return ret; -} - int ipapwd_gen_hashes(struct ipapwd_krbcfg *krbcfg, struct ipapwd_data *data, char *userpw, int is_krb, int is_smb, int is_ipant, Slapi_Value ***svals, @@ -400,6 +205,7 @@ int ipapwd_gen_hashes(struct ipapwd_krbcfg *krbcfg, char **errMesg) { int rc; + char *userpw_uc = NULL; *svals = NULL; *nthash = NULL; @@ -423,10 +229,21 @@ int ipapwd_gen_hashes(struct ipapwd_krbcfg *krbcfg, struct ntlm_keys ntlm; int ret; + userpw_uc = (char *) slapi_utf8StrToUpper((unsigned char *) userpw); + if (!userpw_uc) { + *errMesg = "Failed to generate upper case password\n"; + LOG_FATAL("%s", *errMesg); + rc = LDAP_OPERATIONS_ERROR; + goto done; + } + ret = encode_ntlm_keys(userpw, + userpw_uc, krbcfg->allow_lm_hash, krbcfg->allow_nt_hash, &ntlm); + memset(userpw_uc, 0, strlen(userpw_uc)); + slapi_ch_free_string(&userpw_uc); if (ret) { *errMesg = "Failed to generate NT/LM hashes\n"; LOG_FATAL("%s", *errMesg); diff --git a/util/ipa_pwd.c b/util/ipa_pwd.c index 92fb3b0298418592881d100fb7a9ccfac99fd665..761d1efb8cbcb303d4ec4edd49254b433b048b31 100644 --- a/util/ipa_pwd.c +++ b/util/ipa_pwd.c @@ -20,7 +20,9 @@ * along with this program. If not, see . */ +#ifndef _GNU_SOURCE #define _GNU_SOURCE +#endif #include #include #include diff --git a/util/ipa_pwd.h b/util/ipa_pwd.h index 2e537753e4b6b72cab19f3ce62c04f453e237664..00de889ff53cdc113a6c926e35c87e7b08238e4a 100644 --- a/util/ipa_pwd.h +++ b/util/ipa_pwd.h @@ -71,4 +71,15 @@ int ipapwd_generate_new_history(char *password, char ***new_pwd_history, int *new_pwd_hlen); +struct ntlm_keys { + uint8_t lm[16]; + uint8_t nt[16]; +}; + +int encode_ntlm_keys(char *newPasswd, + char *upperPasswd, + bool do_lm_hash, + bool do_nt_hash, + struct ntlm_keys *keys); + #endif diff --git a/util/ipa_pwd_ntlm.c b/util/ipa_pwd_ntlm.c new file mode 100644 index 0000000000000000000000000000000000000000..a3399b58e006e340677fa43773e6282249858062 --- /dev/null +++ b/util/ipa_pwd_ntlm.c @@ -0,0 +1,213 @@ +/* + * Password related utils for FreeIPA + * + * Authors: Simo Sorce + * + * Copyright (C) 2011,2012 Simo Sorce, Red Hat + * see file 'COPYING' for use and warranty information + * + * This program is free software you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + */ + +#include +#include +#include +#include +#include + +#include "ipa_pwd.h" + +#define KTF_DOS_CHARSET "CP850" /* same default as samba */ +#define KTF_UTF8 "UTF-8" +#define KTF_UCS2 "UCS-2LE" + +static const uint8_t parity_table[128] = { + 1, 2, 4, 7, 8, 11, 13, 14, 16, 19, 21, 22, 25, 26, 28, 31, + 32, 35, 37, 38, 41, 42, 44, 47, 49, 50, 52, 55, 56, 59, 61, 62, + 64, 67, 69, 70, 73, 74, 76, 79, 81, 82, 84, 87, 88, 91, 93, 94, + 97, 98,100,103,104,107,109,110,112,115,117,118,121,122,124,127, + 128,131,133,134,137,138,140,143,145,146,148,151,152,155,157,158, + 161,162,164,167,168,171,173,174,176,179,181,182,185,186,188,191, + 193,194,196,199,200,203,205,206,208,211,213,214,217,218,220,223, + 224,227,229,230,233,234,236,239,241,242,244,247,248,251,253,254 +}; + +static void lm_shuffle(uint8_t *out, uint8_t *in) +{ + out[0] = parity_table[in[0]>>1]; + out[1] = parity_table[((in[0]<<6)|(in[1]>>2)) & 0x7F]; + out[2] = parity_table[((in[1]<<5)|(in[2]>>3)) & 0x7F]; + out[3] = parity_table[((in[2]<<4)|(in[3]>>4)) & 0x7F]; + out[4] = parity_table[((in[3]<<3)|(in[4]>>5)) & 0x7F]; + out[5] = parity_table[((in[4]<<2)|(in[5]>>6)) & 0x7F]; + out[6] = parity_table[((in[5]<<1)|(in[6]>>7)) & 0x7F]; + out[7] = parity_table[in[6] & 0x7F]; +} + +/* create the lm and nt hashes + newPassword: the clear text utf8 password + upperPasswd: upper case version of clear text utf8 password + do_lm_hash: determine if LM hash is generated + do_nt_hash: determine if NT hash is generated + keys[out]: array with generated hashes +*/ +int encode_ntlm_keys(char *newPasswd, + char *upperPasswd, + bool do_lm_hash, + bool do_nt_hash, + struct ntlm_keys *keys) +{ + int ret = 0; + + /* do lanman first */ + if (do_lm_hash) { + iconv_t cd; + size_t cs, il, ol; + char *inc, *outc; + char *asciiPasswd; + DES_key_schedule schedule; + DES_cblock deskey; + DES_cblock magic = "KGS!@#$%"; + + if (upperPasswd == NULL) { + ret = -1; + goto done; + } + il = strlen(upperPasswd); + + /* TODO: must store the dos charset somewhere in the directory */ + cd = iconv_open(KTF_DOS_CHARSET, KTF_UTF8); + if (cd == (iconv_t)(-1)) { + ret = -1; + goto done; + } + + /* an ascii string can only be smaller than or equal to an utf8 one */ + ol = il; + if (ol < 14) ol = 14; + asciiPasswd = calloc(ol+1, 1); + if (!asciiPasswd) { + iconv_close(cd); + ret = -1; + goto done; + } + + inc = upperPasswd; + outc = asciiPasswd; + cs = iconv(cd, &inc, &il, &outc, &ol); + if (cs == -1) { + ret = -1; + free(asciiPasswd); + iconv_close(cd); + goto done; + } + + /* done with these */ + iconv_close(cd); + + /* we are interested only in the first 14 ASCII chars for lanman */ + if (strlen(asciiPasswd) > 14) { + asciiPasswd[14] = '\0'; + } + + /* first half */ + lm_shuffle(deskey, (uint8_t *)asciiPasswd); + + DES_set_key_unchecked(&deskey, &schedule); + DES_ecb_encrypt(&magic, (DES_cblock *)keys->lm, + &schedule, DES_ENCRYPT); + + /* second half */ + lm_shuffle(deskey, (uint8_t *)&asciiPasswd[7]); + + DES_set_key_unchecked(&deskey, &schedule); + DES_ecb_encrypt(&magic, (DES_cblock *)&(keys->lm[8]), + &schedule, DES_ENCRYPT); + + /* done with it */ + free(asciiPasswd); + + } else { + memset(keys->lm, 0, 16); + } + + if (do_nt_hash) { + iconv_t cd; + size_t cs, il, ol, sl; + char *inc, *outc; + char *ucs2Passwd; + MD4_CTX md4ctx; + + /* TODO: must store the dos charset somewhere in the directory */ + cd = iconv_open(KTF_UCS2, KTF_UTF8); + if (cd == (iconv_t)(-1)) { + ret = -1; + goto done; + } + + il = strlen(newPasswd); + + /* an ucs2 string can be at most double than an utf8 one */ + sl = ol = (il+1)*2; + ucs2Passwd = calloc(ol, 1); + if (!ucs2Passwd) { + ret = -1; + iconv_close(cd); + goto done; + } + + inc = newPasswd; + outc = ucs2Passwd; + cs = iconv(cd, &inc, &il, &outc, &ol); + if (cs == -1) { + ret = -1; + free(ucs2Passwd); + iconv_close(cd); + goto done; + } + + /* done with it */ + iconv_close(cd); + + /* get the final ucs2 string length */ + sl -= ol; + + ret = MD4_Init(&md4ctx); + if (ret == 0) { + ret = -1; + free(ucs2Passwd); + goto done; + } + ret = MD4_Update(&md4ctx, ucs2Passwd, sl); + if (ret == 0) { + ret = -1; + free(ucs2Passwd); + goto done; + } + ret = MD4_Final(keys->nt, &md4ctx); + if (ret == 0) { + ret = -1; + free(ucs2Passwd); + goto done; + } + + } else { + memset(keys->nt, 0, 16); + } + + ret = 0; + +done: + return ret; +} -- 1.7.11.4 -------------- next part -------------- From f7e1cd0d664bd1779e5b40c189da69bc88d7a1b2 Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Fri, 24 Aug 2012 14:46:29 +0200 Subject: [PATCH 44/59] ipasam: remove nt_lm_owf_gen() and dependency to libcliauth.so --- daemons/ipa-sam/ipa_sam.c | 71 +++++++++++++++++++++++++++++++++++++++-------- 1 Datei ge?ndert, 59 Zeilen hinzugef?gt(+), 12 Zeilen entfernt(-) diff --git a/daemons/ipa-sam/ipa_sam.c b/daemons/ipa-sam/ipa_sam.c index 8a4a08bc7a5951553a463805a8aedb82ee887936..aacae68a4d1c23af3f47ad7f4a7f115c90ea8626 100644 --- a/daemons/ipa-sam/ipa_sam.c +++ b/daemons/ipa-sam/ipa_sam.c @@ -30,6 +30,7 @@ #include #include #include "ipa_krb5.h" +#include "ipa_pwd.h" /* from drsblobs.h */ struct AuthInfoNone { @@ -81,7 +82,6 @@ struct trustAuthInOutBlob { enum ndr_err_code ndr_pull_trustAuthInOutBlob(struct ndr_pull *ndr, int ndr_flags, struct trustAuthInOutBlob *r); /*available in libndr-samba.so */ bool fetch_ldap_pw(char **dn, char** pw); /* available in libpdb.so */ -void nt_lm_owf_gen(const char *pwd, uint8_t nt_p16[16], uint8_t p16[16]); /* available in libcliauth.so */ bool sid_check_is_builtin(const struct dom_sid *sid); /* available in libpdb.so */ /* available in libpdb.so, renamed from sid_check_is_domain() in c43505b621725c9a754f0ee98318d451b093f2ed */ bool sid_check_is_our_sam(const struct dom_sid *sid); @@ -2348,9 +2348,14 @@ static bool init_sam_from_td(struct samu *user, struct pdb_trusted_domain *td, NTSTATUS status; struct dom_sid u_sid; char *name; - uint8_t smblmpwd[LM_HASH_LEN]; - uint8_t smbntpwd[NT_HASH_LEN]; - char *trustpw; + char *trustpw = NULL; + char *trustpw_utf8 = NULL; + char *trustpw_utf8_uc = NULL; + char *tmp_str = NULL; + int ret; + struct ntlm_keys ntlm_keys; + size_t converted_size; + bool res; if (!pdb_set_acct_ctrl(user, ACB_DOMTRUST | ACB_TRUSTED_FOR_DELEGATION, PDB_SET)) { @@ -2387,17 +2392,59 @@ static bool init_sam_from_td(struct samu *user, struct pdb_trusted_domain *td, if (!NT_STATUS_IS_OK(status)) { return false; } - nt_lm_owf_gen(trustpw, smbntpwd, smblmpwd); - memset(trustpw, 0, strlen(trustpw)); - talloc_free(trustpw); - if (!pdb_set_lanman_passwd(user, smblmpwd, PDB_SET)) { - return false; + + if (!push_utf8_talloc(user, &trustpw_utf8, trustpw, &converted_size)) { + res = false; + goto done; + } + + tmp_str = talloc_strdup_upper(user, trustpw); + if (tmp_str == NULL) { + res = false; + goto done; + } + + if (!push_utf8_talloc(user, &trustpw_utf8_uc, tmp_str, &converted_size)) { + res = false; + goto done; + } + + ret = encode_ntlm_keys(trustpw_utf8, trustpw_utf8_uc, true, true, + &ntlm_keys); + if (ret != 0) { + res = false; + goto done; + } + + if (!pdb_set_lanman_passwd(user, ntlm_keys.lm, PDB_SET)) { + res = false; + goto done; + } + if (!pdb_set_nt_passwd(user, ntlm_keys.nt, PDB_SET)) { + res = false; + goto done; + } + + res = true; +done: + if (trustpw != NULL) { + memset(trustpw, 0, strlen(trustpw)); + talloc_free(trustpw); + } + if (trustpw_utf8 != NULL) { + memset(trustpw_utf8, 0, strlen(trustpw_utf8)); + talloc_free(trustpw_utf8); + } + if (tmp_str != NULL) { + memset(tmp_str, 0, strlen(tmp_str)); + talloc_free(tmp_str); } - if (!pdb_set_nt_passwd(user, smbntpwd, PDB_SET)) { - return false; + if (trustpw_utf8_uc != NULL) { + memset(trustpw_utf8_uc, 0, strlen(trustpw_utf8_uc)); + talloc_free(trustpw_utf8_uc); } - return true; + return res; } static bool ipasam_nthash_retrieve(struct ldapsam_privates *ldap_state, -- 1.7.11.4 -------------- next part -------------- From 002f63ac97a5936e471993b114b41e675539145b Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Fri, 24 Aug 2012 15:01:57 +0200 Subject: [PATCH 45/59] ipasam: remove sid_peek_rid() --- daemons/ipa-sam/ipa_sam.c | 29 ++++++++++++++++++----------- 1 Datei ge?ndert, 18 Zeilen hinzugef?gt(+), 11 Zeilen entfernt(-) diff --git a/daemons/ipa-sam/ipa_sam.c b/daemons/ipa-sam/ipa_sam.c index aacae68a4d1c23af3f47ad7f4a7f115c90ea8626..ce8f623bdc5af9cdedb57e92bdee421326ce68bd 100644 --- a/daemons/ipa-sam/ipa_sam.c +++ b/daemons/ipa-sam/ipa_sam.c @@ -91,7 +91,6 @@ void sid_copy(struct dom_sid *dst, const struct dom_sid *src); /* available in l bool sid_linearize(char *outbuf, size_t len, const struct dom_sid *sid); /* available in libsmbconf.so */ bool string_to_sid(struct dom_sid *sidout, const char *sidstr); /* available in libsecurity.so */ bool sid_compose(struct dom_sid *dst, const struct dom_sid *domain_sid, uint32_t rid); /* available in libsecurity.so */ -bool sid_peek_rid(const struct dom_sid *sid, uint32_t *rid); /* available in libsecurity.so */ int dom_sid_compare_domain(const struct dom_sid *sid1, const struct dom_sid *sid2); /* available in libsecurity.so */ char *sid_string_talloc(TALLOC_CTX *mem_ctx, const struct dom_sid *sid); /* available in libsmbconf.so */ char *sid_string_dbg(const struct dom_sid *sid); /* available in libsmbconf.so */ @@ -246,35 +245,43 @@ static bool ldapsam_extract_rid_from_entry(LDAP *ldap_struct, const struct dom_sid *domain_sid, uint32_t *rid) { - char *str; + char *str = NULL; struct dom_sid sid; + bool res = false; str = get_single_attribute(NULL, ldap_struct, entry, LDAP_ATTRIBUTE_SID); if (str == NULL) { DEBUG(10, ("Could not find SID attribute\n")); - return false; + res = false; + goto done; } if (!string_to_sid(&sid, str)) { - talloc_free(str); DEBUG(10, ("Could not convert string %s to sid\n", str)); - return false; + res = false; + goto done; } - talloc_free(str); if (dom_sid_compare_domain(&sid, domain_sid) != 0) { DEBUG(10, ("SID %s is not in expected domain %s\n", str, sid_string_dbg(domain_sid))); - return false; + res = false; + goto done; } - if (!sid_peek_rid(&sid, rid)) { - DEBUG(10, ("Could not peek into RID\n")); - return false; + if (sid.num_auths <= 0) { + DEBUG(10, ("Invalid num_auths in SID %s.\n", str)); + res = false; + goto done; } - return true; + *rid = sid.sub_auths[sid.num_auths - 1]; + + res = true; +done: + talloc_free(str); + return res; } static NTSTATUS ldapsam_lookup_rids(struct pdb_methods *methods, -- 1.7.11.4 -------------- next part -------------- From 6546840d06bb1cac9faf260a2065a27a9584015a Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Fri, 24 Aug 2012 15:12:21 +0200 Subject: [PATCH 46/59] ipasam: replace strnequal() --- daemons/ipa-sam/ipa_sam.c | 17 ++++++++++++++++- 1 Datei ge?ndert, 16 Zeilen hinzugef?gt(+), 1 Zeile entfernt(-) diff --git a/daemons/ipa-sam/ipa_sam.c b/daemons/ipa-sam/ipa_sam.c index ce8f623bdc5af9cdedb57e92bdee421326ce68bd..be033476d4cbeff1e91019986aebf9504329ec4f 100644 --- a/daemons/ipa-sam/ipa_sam.c +++ b/daemons/ipa-sam/ipa_sam.c @@ -95,7 +95,6 @@ int dom_sid_compare_domain(const struct dom_sid *sid1, const struct dom_sid *sid char *sid_string_talloc(TALLOC_CTX *mem_ctx, const struct dom_sid *sid); /* available in libsmbconf.so */ char *sid_string_dbg(const struct dom_sid *sid); /* available in libsmbconf.so */ bool is_null_sid(const struct dom_sid *sid); /* available in libsecurity.so */ -bool strnequal(const char *s1,const char *s2,size_t n); /* available in libutil_str.so */ bool trim_char(char *s,char cfront,char cback); /* available in libutil_str.so */ bool sid_peek_check_rid(const struct dom_sid *exp_dom_sid, const struct dom_sid *sid, uint32_t *rid); /* available in libsecurity.so */ char *escape_ldap_string(TALLOC_CTX *mem_ctx, const char *s); /* available in libsmbconf.so */ @@ -169,6 +168,22 @@ struct ipasam_privates { char *client_princ; }; +static bool strnequal(const char *s1, const char *s2, size_t n) { + if (s1 == s2) { + return true; + } + + if (s1 == NULL || s2 == NULL || n == 0) { + return false; + } + + if (strncasecmp(s1, s2, n) == 0) { + return true; + } + + return false; +} + static LDAP *priv2ld(struct ldapsam_privates *priv) { return priv->smbldap_state->ldap_struct; -- 1.7.11.4 -------------- next part -------------- From b4e4d4c65f56a856e867db62955fb43805ec2963 Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Fri, 24 Aug 2012 18:02:02 +0200 Subject: [PATCH 47/59] ipasam: remove strlower_m() --- daemons/ipa-sam/ipa_sam.c | 4 +--- 1 Datei ge?ndert, 1 Zeile hinzugef?gt(+), 3 Zeilen entfernt(-) diff --git a/daemons/ipa-sam/ipa_sam.c b/daemons/ipa-sam/ipa_sam.c index be033476d4cbeff1e91019986aebf9504329ec4f..331df733323b34437cfff423ed72c88845976efc 100644 --- a/daemons/ipa-sam/ipa_sam.c +++ b/daemons/ipa-sam/ipa_sam.c @@ -85,7 +85,6 @@ bool fetch_ldap_pw(char **dn, char** pw); /* available in libpdb.so */ bool sid_check_is_builtin(const struct dom_sid *sid); /* available in libpdb.so */ /* available in libpdb.so, renamed from sid_check_is_domain() in c43505b621725c9a754f0ee98318d451b093f2ed */ bool sid_check_is_our_sam(const struct dom_sid *sid); -void strlower_m(char *s); /* available in libutil_str.so */ char *talloc_asprintf_strupper_m(TALLOC_CTX *t, const char *fmt, ...); /* available in libutil_str.so */ void sid_copy(struct dom_sid *dst, const struct dom_sid *src); /* available in libsecurity.so */ bool sid_linearize(char *outbuf, size_t len, const struct dom_sid *sid); /* available in libsmbconf.so */ @@ -2910,11 +2909,10 @@ static struct pdb_domain_info *pdb_ipasam_get_domain_info(struct pdb_methods *pd } /* TODO: read dns_domain, dns_forest and guid from LDAP */ - info->dns_domain = talloc_strdup(info, ldap_state->ipasam_privates->realm); + info->dns_domain = strlower_talloc(info, ldap_state->ipasam_privates->realm); if (info->dns_domain == NULL) { goto fail; } - strlower_m(info->dns_domain); info->dns_forest = talloc_strdup(info, info->dns_domain); /* we expect a domain SID to have 4 sub IDs */ -- 1.7.11.4 -------------- next part -------------- From fb3fc61f3ff6e0c7827dc0b944ec6f3c7c308fc4 Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Fri, 24 Aug 2012 18:22:34 +0200 Subject: [PATCH 48/59] ipasam: remove talloc_asprintf_strupper_m() --- daemons/ipa-sam/ipa_sam.c | 11 ++++++++--- 1 Datei ge?ndert, 8 Zeilen hinzugef?gt(+), 3 Zeilen entfernt(-) diff --git a/daemons/ipa-sam/ipa_sam.c b/daemons/ipa-sam/ipa_sam.c index 331df733323b34437cfff423ed72c88845976efc..c3eff47d3abfdbe40f4f49448a968f9bef4132d1 100644 --- a/daemons/ipa-sam/ipa_sam.c +++ b/daemons/ipa-sam/ipa_sam.c @@ -85,7 +85,6 @@ bool fetch_ldap_pw(char **dn, char** pw); /* available in libpdb.so */ bool sid_check_is_builtin(const struct dom_sid *sid); /* available in libpdb.so */ /* available in libpdb.so, renamed from sid_check_is_domain() in c43505b621725c9a754f0ee98318d451b093f2ed */ bool sid_check_is_our_sam(const struct dom_sid *sid); -char *talloc_asprintf_strupper_m(TALLOC_CTX *t, const char *fmt, ...); /* available in libutil_str.so */ void sid_copy(struct dom_sid *dst, const struct dom_sid *src); /* available in libsecurity.so */ bool sid_linearize(char *outbuf, size_t len, const struct dom_sid *sid); /* available in libsmbconf.so */ bool string_to_sid(struct dom_sid *sidout, const char *sidstr); /* available in libsecurity.so */ @@ -3169,8 +3168,14 @@ static NTSTATUS ipasam_get_realm(struct ldapsam_privates *ldap_state, #define SECRETS_DOMAIN_SID "SECRETS/SID" static char *sec_key(TALLOC_CTX *mem_ctx, const char *d) { - return talloc_asprintf_strupper_m(mem_ctx, "%s/%s", - SECRETS_DOMAIN_SID, d); + char *tmp; + char *res; + + tmp = talloc_asprintf(mem_ctx, "%s/%s", SECRETS_DOMAIN_SID, d); + res = talloc_strdup_upper(mem_ctx, tmp); + talloc_free(tmp); + + return res; } static NTSTATUS save_sid_to_secret(struct ldapsam_privates *ldap_state) -- 1.7.11.4 -------------- next part -------------- From 59a37e80c1285f99290bfb28fd05a35d29e3622a Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Mon, 27 Aug 2012 13:42:26 +0200 Subject: [PATCH 49/59] ipasam: replace sid_copy() --- daemons/ipa-sam/ipa_sam.c | 16 +++++++++++++++- 1 Datei ge?ndert, 15 Zeilen hinzugef?gt(+), 1 Zeile entfernt(-) diff --git a/daemons/ipa-sam/ipa_sam.c b/daemons/ipa-sam/ipa_sam.c index c3eff47d3abfdbe40f4f49448a968f9bef4132d1..e7774f5b3d14ce6ed4ac87eda5af5dda79813e20 100644 --- a/daemons/ipa-sam/ipa_sam.c +++ b/daemons/ipa-sam/ipa_sam.c @@ -85,7 +85,6 @@ bool fetch_ldap_pw(char **dn, char** pw); /* available in libpdb.so */ bool sid_check_is_builtin(const struct dom_sid *sid); /* available in libpdb.so */ /* available in libpdb.so, renamed from sid_check_is_domain() in c43505b621725c9a754f0ee98318d451b093f2ed */ bool sid_check_is_our_sam(const struct dom_sid *sid); -void sid_copy(struct dom_sid *dst, const struct dom_sid *src); /* available in libsecurity.so */ bool sid_linearize(char *outbuf, size_t len, const struct dom_sid *sid); /* available in libsmbconf.so */ bool string_to_sid(struct dom_sid *sidout, const char *sidstr); /* available in libsecurity.so */ bool sid_compose(struct dom_sid *dst, const struct dom_sid *domain_sid, uint32_t rid); /* available in libsecurity.so */ @@ -166,6 +165,21 @@ struct ipasam_privates { char *client_princ; }; +static void sid_copy(struct dom_sid *dst, const struct dom_sid *src) +{ + size_t c; + + memset(dst, 0, sizeof(*dst)); + + dst->sid_rev_num = src->sid_rev_num; + dst->num_auths = src->num_auths; + memcpy(&dst->id_auth[0], &src->id_auth[0], sizeof(src->id_auth)); + + for (c = 0; c < src->num_auths; c++) { + dst->sub_auths[c] = src->sub_auths[c]; + } +} + static bool strnequal(const char *s1, const char *s2, size_t n) { if (s1 == s2) { return true; -- 1.7.11.4 -------------- next part -------------- From 0e260f624cda9dc424da7cec6da23a9aff644177 Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Mon, 27 Aug 2012 13:54:08 +0200 Subject: [PATCH 50/59] ipasam: replace sid_compose() --- daemons/ipa-sam/ipa_sam.c | 15 ++++++++++++++- 1 Datei ge?ndert, 14 Zeilen hinzugef?gt(+), 1 Zeile entfernt(-) diff --git a/daemons/ipa-sam/ipa_sam.c b/daemons/ipa-sam/ipa_sam.c index e7774f5b3d14ce6ed4ac87eda5af5dda79813e20..3f656246f6f0f5f59a2f92d5ee53b57be78a17dd 100644 --- a/daemons/ipa-sam/ipa_sam.c +++ b/daemons/ipa-sam/ipa_sam.c @@ -87,7 +87,6 @@ bool sid_check_is_builtin(const struct dom_sid *sid); /* available in libpdb.so bool sid_check_is_our_sam(const struct dom_sid *sid); bool sid_linearize(char *outbuf, size_t len, const struct dom_sid *sid); /* available in libsmbconf.so */ bool string_to_sid(struct dom_sid *sidout, const char *sidstr); /* available in libsecurity.so */ -bool sid_compose(struct dom_sid *dst, const struct dom_sid *domain_sid, uint32_t rid); /* available in libsecurity.so */ int dom_sid_compare_domain(const struct dom_sid *sid1, const struct dom_sid *sid2); /* available in libsecurity.so */ char *sid_string_talloc(TALLOC_CTX *mem_ctx, const struct dom_sid *sid); /* available in libsmbconf.so */ char *sid_string_dbg(const struct dom_sid *sid); /* available in libsmbconf.so */ @@ -180,6 +179,20 @@ static void sid_copy(struct dom_sid *dst, const struct dom_sid *src) } } +static bool sid_compose(struct dom_sid *dst, const struct dom_sid *dom_sid, + uint32_t rid) +{ + if (dom_sid->num_auths >= 15) { + return false; + } + + sid_copy(dst, dom_sid); + + dst->sub_auths[dst->num_auths++] = rid; + + return true; +} + static bool strnequal(const char *s1, const char *s2, size_t n) { if (s1 == s2) { return true; -- 1.7.11.4 -------------- next part -------------- From 35a1e7d830f3998ee40ec8a5beed5193fcb6097d Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Mon, 27 Aug 2012 14:03:43 +0200 Subject: [PATCH 51/59] ipasam: Replace is_null_sid() --- daemons/ipa-sam/ipa_sam.c | 25 ++++++++++++++++++++++++- 1 Datei ge?ndert, 24 Zeilen hinzugef?gt(+), 1 Zeile entfernt(-) diff --git a/daemons/ipa-sam/ipa_sam.c b/daemons/ipa-sam/ipa_sam.c index 3f656246f6f0f5f59a2f92d5ee53b57be78a17dd..879e60ba49bc7625efbf39aec45ed19b4becfd29 100644 --- a/daemons/ipa-sam/ipa_sam.c +++ b/daemons/ipa-sam/ipa_sam.c @@ -90,7 +90,6 @@ bool string_to_sid(struct dom_sid *sidout, const char *sidstr); /* available in int dom_sid_compare_domain(const struct dom_sid *sid1, const struct dom_sid *sid2); /* available in libsecurity.so */ char *sid_string_talloc(TALLOC_CTX *mem_ctx, const struct dom_sid *sid); /* available in libsmbconf.so */ char *sid_string_dbg(const struct dom_sid *sid); /* available in libsmbconf.so */ -bool is_null_sid(const struct dom_sid *sid); /* available in libsecurity.so */ bool trim_char(char *s,char cfront,char cback); /* available in libutil_str.so */ bool sid_peek_check_rid(const struct dom_sid *exp_dom_sid, const struct dom_sid *sid, uint32_t *rid); /* available in libsecurity.so */ char *escape_ldap_string(TALLOC_CTX *mem_ctx, const char *s); /* available in libsmbconf.so */ @@ -193,6 +192,30 @@ static bool sid_compose(struct dom_sid *dst, const struct dom_sid *dom_sid, return true; } +static bool is_null_sid(const struct dom_sid *sid) +{ + size_t c; + + if (sid->sid_rev_num != 0 || sid->num_auths != 0) { + return false; + } + + for (c = 0; c < 6; c++) { + if (sid->id_auth[c] != 0) { + return false; + } + } + + for (c = 0; c < 15; c++) { + if (sid->sub_auths[c] != 0) { + return false; + } + } + + return true; +} + + static bool strnequal(const char *s1, const char *s2, size_t n) { if (s1 == s2) { return true; -- 1.7.11.4 -------------- next part -------------- From d980d03d52138108b30b269daff0a11c82b8499d Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Mon, 27 Aug 2012 16:21:02 +0200 Subject: [PATCH 52/59] ipasam: Replace dom_sid_compare_domain() --- daemons/ipa-sam/ipa_sam.c | 28 +++++++++++++++++++++++++++- 1 Datei ge?ndert, 27 Zeilen hinzugef?gt(+), 1 Zeile entfernt(-) diff --git a/daemons/ipa-sam/ipa_sam.c b/daemons/ipa-sam/ipa_sam.c index 879e60ba49bc7625efbf39aec45ed19b4becfd29..5ef27dc4d75bcc5eb504cbbc1883fb98374f9c2b 100644 --- a/daemons/ipa-sam/ipa_sam.c +++ b/daemons/ipa-sam/ipa_sam.c @@ -87,7 +87,6 @@ bool sid_check_is_builtin(const struct dom_sid *sid); /* available in libpdb.so bool sid_check_is_our_sam(const struct dom_sid *sid); bool sid_linearize(char *outbuf, size_t len, const struct dom_sid *sid); /* available in libsmbconf.so */ bool string_to_sid(struct dom_sid *sidout, const char *sidstr); /* available in libsecurity.so */ -int dom_sid_compare_domain(const struct dom_sid *sid1, const struct dom_sid *sid2); /* available in libsecurity.so */ char *sid_string_talloc(TALLOC_CTX *mem_ctx, const struct dom_sid *sid); /* available in libsmbconf.so */ char *sid_string_dbg(const struct dom_sid *sid); /* available in libsmbconf.so */ bool trim_char(char *s,char cfront,char cback); /* available in libutil_str.so */ @@ -215,6 +214,33 @@ static bool is_null_sid(const struct dom_sid *sid) return true; } +static int dom_sid_compare_domain(const struct dom_sid *sid1, + const struct dom_sid *sid2) +{ + size_t c; + size_t n_sub_auths; + + if (sid1->sid_rev_num != sid2->sid_rev_num) { + return sid1->sid_rev_num - sid2->sid_rev_num; + } + + for (c = 0; c < 6; c++) { + if (sid1->id_auth[c] != sid2->id_auth[c]) { + return sid1->id_auth[c] - sid2->id_auth[c]; + } + } + + n_sub_auths = (sid1->num_auths < sid2->num_auths) ? sid1->num_auths : + sid2->num_auths; + + for (c = 0; c < n_sub_auths; c++) { + if (sid1->sub_auths[c] != sid2->sub_auths[c]) { + return sid1->sub_auths[c] - sid2->sub_auths[c]; + } + } + + return 0; +} static bool strnequal(const char *s1, const char *s2, size_t n) { if (s1 == s2) { -- 1.7.11.4 -------------- next part -------------- From 658e275ac2480964ac8a5552ea82a88d0b5024a7 Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Mon, 27 Aug 2012 16:28:48 +0200 Subject: [PATCH 53/59] ipasam: Replace sid_check_is_our_sam() --- daemons/ipa-sam/ipa_sam.c | 3 +-- 1 Datei ge?ndert, 1 Zeile hinzugef?gt(+), 2 Zeilen entfernt(-) diff --git a/daemons/ipa-sam/ipa_sam.c b/daemons/ipa-sam/ipa_sam.c index 5ef27dc4d75bcc5eb504cbbc1883fb98374f9c2b..d666664a74cc914bce922f1db9a339db939eb88d 100644 --- a/daemons/ipa-sam/ipa_sam.c +++ b/daemons/ipa-sam/ipa_sam.c @@ -84,7 +84,6 @@ enum ndr_err_code ndr_pull_trustAuthInOutBlob(struct ndr_pull *ndr, int ndr_flag bool fetch_ldap_pw(char **dn, char** pw); /* available in libpdb.so */ bool sid_check_is_builtin(const struct dom_sid *sid); /* available in libpdb.so */ /* available in libpdb.so, renamed from sid_check_is_domain() in c43505b621725c9a754f0ee98318d451b093f2ed */ -bool sid_check_is_our_sam(const struct dom_sid *sid); bool sid_linearize(char *outbuf, size_t len, const struct dom_sid *sid); /* available in libsmbconf.so */ bool string_to_sid(struct dom_sid *sidout, const char *sidstr); /* available in libsecurity.so */ char *sid_string_talloc(TALLOC_CTX *mem_ctx, const struct dom_sid *sid); /* available in libsmbconf.so */ @@ -398,7 +397,7 @@ static NTSTATUS ldapsam_lookup_rids(struct pdb_methods *methods, } if (!sid_check_is_builtin(domain_sid) && - !sid_check_is_our_sam(domain_sid)) { + dom_sid_compare_domain(&ldap_state->domain_sid, domain_sid) != 0) { result = NT_STATUS_INVALID_PARAMETER; goto done; } -- 1.7.11.4 -------------- next part -------------- From 9f696010e4b4b5cdea3a6d45560eabc31e48d6e6 Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Mon, 27 Aug 2012 17:19:26 +0200 Subject: [PATCH 54/59] ipasam: Replace sid_peek_check_rid() --- daemons/ipa-sam/ipa_sam.c | 18 +++++++++++++++++- 1 Datei ge?ndert, 17 Zeilen hinzugef?gt(+), 1 Zeile entfernt(-) diff --git a/daemons/ipa-sam/ipa_sam.c b/daemons/ipa-sam/ipa_sam.c index d666664a74cc914bce922f1db9a339db939eb88d..199c306a7e2402b3c2dd1d9ca0154d59ff5ac335 100644 --- a/daemons/ipa-sam/ipa_sam.c +++ b/daemons/ipa-sam/ipa_sam.c @@ -89,7 +89,6 @@ bool string_to_sid(struct dom_sid *sidout, const char *sidstr); /* available in char *sid_string_talloc(TALLOC_CTX *mem_ctx, const struct dom_sid *sid); /* available in libsmbconf.so */ char *sid_string_dbg(const struct dom_sid *sid); /* available in libsmbconf.so */ bool trim_char(char *s,char cfront,char cback); /* available in libutil_str.so */ -bool sid_peek_check_rid(const struct dom_sid *exp_dom_sid, const struct dom_sid *sid, uint32_t *rid); /* available in libsecurity.so */ char *escape_ldap_string(TALLOC_CTX *mem_ctx, const char *s); /* available in libsmbconf.so */ extern const struct dom_sid global_sid_Builtin; /* available in libsecurity.so */ bool secrets_store(const char *key, const void *data, size_t size); /* available in libpdb.so */ @@ -241,6 +240,23 @@ static int dom_sid_compare_domain(const struct dom_sid *sid1, return 0; } +static bool sid_peek_check_rid(const struct dom_sid *exp_dom_sid, + const struct dom_sid *sid, uint32_t *rid) +{ + if((exp_dom_sid->num_auths + 1) != sid->num_auths || + sid->num_auths <= 0) { + return false; + } + + if (dom_sid_compare_domain(exp_dom_sid, sid) != 0) { + return false; + } + + *rid = sid->sub_auths[sid->num_auths - 1]; + + return true; +} + static bool strnequal(const char *s1, const char *s2, size_t n) { if (s1 == s2) { return true; -- 1.7.11.4 -------------- next part -------------- From 10b5780ebf6d2329a7269e9e7327555f1a01026e Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Mon, 27 Aug 2012 17:26:57 +0200 Subject: [PATCH 55/59] ipasam: Replace global_sid_Builtin --- daemons/ipa-sam/ipa_sam.c | 4 +++- 1 Datei ge?ndert, 3 Zeilen hinzugef?gt(+), 1 Zeile entfernt(-) diff --git a/daemons/ipa-sam/ipa_sam.c b/daemons/ipa-sam/ipa_sam.c index 199c306a7e2402b3c2dd1d9ca0154d59ff5ac335..b3be5a67921045e5b704b2a20098721aa0c8fa5b 100644 --- a/daemons/ipa-sam/ipa_sam.c +++ b/daemons/ipa-sam/ipa_sam.c @@ -90,7 +90,6 @@ char *sid_string_talloc(TALLOC_CTX *mem_ctx, const struct dom_sid *sid); /* avai char *sid_string_dbg(const struct dom_sid *sid); /* available in libsmbconf.so */ bool trim_char(char *s,char cfront,char cback); /* available in libutil_str.so */ char *escape_ldap_string(TALLOC_CTX *mem_ctx, const char *s); /* available in libsmbconf.so */ -extern const struct dom_sid global_sid_Builtin; /* available in libsecurity.so */ bool secrets_store(const char *key, const void *data, size_t size); /* available in libpdb.so */ #define LDAP_PAGE_SIZE 1024 @@ -150,6 +149,9 @@ bool secrets_store(const char *key, const void *data, size_t size); /* available #define HAS_POSIXGROUP (1<<8) #define HAS_KRB_TICKET_POLICY_AUX (1<<9) +const struct dom_sid global_sid_Builtin = { 1, 1, {0,0,0,0,0,5}, + {32,0,0,0,0,0,0,0,0,0,0,0,0,0,0}}; + struct ipasam_privates { char *realm; char *base_dn; -- 1.7.11.4 -------------- next part -------------- From 15921a09d48b01d5234e3d8748dce7e5ed89cbde Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Mon, 27 Aug 2012 21:26:30 +0200 Subject: [PATCH 56/59] ipasam: add libsss_idmap context and replace string_to_sid() --- daemons/ipa-sam/Makefile.am | 2 + daemons/ipa-sam/ipa_sam.c | 118 ++++++++++++++++++++++++++++++++------------ 2 Dateien ge?ndert, 89 Zeilen hinzugef?gt(+), 31 Zeilen entfernt(-) diff --git a/daemons/ipa-sam/Makefile.am b/daemons/ipa-sam/Makefile.am index 11179276c17f9b0415fc5ad944b88bcbbb0529c3..53c8f47bbfe927caf785a5529fb5d6e2dcbc7525 100644 --- a/daemons/ipa-sam/Makefile.am +++ b/daemons/ipa-sam/Makefile.am @@ -27,6 +27,7 @@ INCLUDES = \ $(TALLOC_CFLAGS) \ $(SAMBAUTIL_CFLAGS) \ $(NDR_CFLAGS) \ + $(SSSIDMAP_CFLAGS) \ $(NULL) plugindir = $(libdir)/samba/pdb @@ -52,6 +53,7 @@ ipasam_la_LIBADD = \ $(SAMBAUTIL_LIBS) \ $(NDR_LIBS) \ $(SAMBA40EXTRA_LIBS) \ + $(SSSIDMAP_LIBS) \ $(NULL) EXTRA_DIST = \ diff --git a/daemons/ipa-sam/ipa_sam.c b/daemons/ipa-sam/ipa_sam.c index b3be5a67921045e5b704b2a20098721aa0c8fa5b..58a598f5aaf88dfeb95a76adef3a79b085638cd0 100644 --- a/daemons/ipa-sam/ipa_sam.c +++ b/daemons/ipa-sam/ipa_sam.c @@ -29,6 +29,7 @@ #include #include +#include #include "ipa_krb5.h" #include "ipa_pwd.h" @@ -85,7 +86,6 @@ bool fetch_ldap_pw(char **dn, char** pw); /* available in libpdb.so */ bool sid_check_is_builtin(const struct dom_sid *sid); /* available in libpdb.so */ /* available in libpdb.so, renamed from sid_check_is_domain() in c43505b621725c9a754f0ee98318d451b093f2ed */ bool sid_linearize(char *outbuf, size_t len, const struct dom_sid *sid); /* available in libsmbconf.so */ -bool string_to_sid(struct dom_sid *sidout, const char *sidstr); /* available in libsecurity.so */ char *sid_string_talloc(TALLOC_CTX *mem_ctx, const struct dom_sid *sid); /* available in libsmbconf.so */ char *sid_string_dbg(const struct dom_sid *sid); /* available in libsmbconf.so */ bool trim_char(char *s,char cfront,char cback); /* available in libutil_str.so */ @@ -160,8 +160,19 @@ struct ipasam_privates { char *fallback_primary_group; char *server_princ; char *client_princ; + struct sss_idmap_ctx *idmap_ctx; }; +static void *idmap_talloc(size_t size, void *pvt) +{ + return talloc_size(pvt, size); +} + +static void idmap_talloc_free(void *ptr, void *pvt) +{ + talloc_free(ptr); +} + static void sid_copy(struct dom_sid *dst, const struct dom_sid *src) { size_t c; @@ -348,12 +359,14 @@ done: static bool ldapsam_extract_rid_from_entry(LDAP *ldap_struct, LDAPMessage *entry, + struct sss_idmap_ctx *idmap_ctx, const struct dom_sid *domain_sid, uint32_t *rid) { char *str = NULL; - struct dom_sid sid; + struct dom_sid *sid = NULL; bool res = false; + enum idmap_error_code err; str = get_single_attribute(NULL, ldap_struct, entry, LDAP_ATTRIBUTE_SID); @@ -363,29 +376,31 @@ static bool ldapsam_extract_rid_from_entry(LDAP *ldap_struct, goto done; } - if (!string_to_sid(&sid, str)) { + err = sss_idmap_sid_to_smb_sid(idmap_ctx, str, &sid); + if (err != IDMAP_SUCCESS) { DEBUG(10, ("Could not convert string %s to sid\n", str)); res = false; goto done; } - if (dom_sid_compare_domain(&sid, domain_sid) != 0) { + if (dom_sid_compare_domain(sid, domain_sid) != 0) { DEBUG(10, ("SID %s is not in expected domain %s\n", str, sid_string_dbg(domain_sid))); res = false; goto done; } - if (sid.num_auths <= 0) { + if (sid->num_auths <= 0) { DEBUG(10, ("Invalid num_auths in SID %s.\n", str)); res = false; goto done; } - *rid = sid.sub_auths[sid.num_auths - 1]; + *rid = sid->sub_auths[sid->num_auths - 1]; res = true; done: + talloc_free(sid); talloc_free(str); return res; } @@ -479,7 +494,9 @@ static NTSTATUS ldapsam_lookup_rids(struct pdb_methods *methods, int rid_index; const char *name; - if (!ldapsam_extract_rid_from_entry(ld, entry, domain_sid, + if (!ldapsam_extract_rid_from_entry(ld, entry, + ldap_state->ipasam_privates->idmap_ctx, + domain_sid, &rid)) { DEBUG(2, ("Could not find sid from ldap entry\n")); continue; @@ -564,8 +581,9 @@ static NTSTATUS ldapsam_lookup_rids(struct pdb_methods *methods, DEBUG(2, ("Rejecting invalid group mapping entry %s\n", dn)); } - if (!ldapsam_extract_rid_from_entry(ld, entry, domain_sid, - &rid)) { + if (!ldapsam_extract_rid_from_entry(ld, entry, + ldap_state->ipasam_privates->idmap_ctx, + domain_sid, &rid)) { DEBUG(2, ("Could not find sid from ldap entry %s\n", dn)); continue; } @@ -718,8 +736,9 @@ static bool ldapsam_uid_to_sid(struct pdb_methods *methods, uid_t uid, LDAPMessage *entry = NULL; bool ret = false; char *user_sid_string; - struct dom_sid user_sid; + struct dom_sid *user_sid = NULL; int rc; + enum idmap_error_code err; TALLOC_CTX *tmp_ctx = talloc_stackframe(); filter = talloc_asprintf(tmp_ctx, @@ -757,17 +776,20 @@ static bool ldapsam_uid_to_sid(struct pdb_methods *methods, uid_t uid, goto done; } - if (!string_to_sid(&user_sid, user_sid_string)) { + err = sss_idmap_sid_to_smb_sid(priv->ipasam_privates->idmap_ctx, + user_sid_string, &user_sid); + if (err != IDMAP_SUCCESS) { DEBUG(3, ("Error calling sid_string_talloc for sid '%s'\n", user_sid_string)); goto done; } - sid_copy(sid, &user_sid); + sid_copy(sid, user_sid); ret = true; - done: +done: + talloc_free(user_sid); TALLOC_FREE(tmp_ctx); return ret; } @@ -783,8 +805,9 @@ static bool ldapsam_gid_to_sid(struct pdb_methods *methods, gid_t gid, LDAPMessage *entry = NULL; bool ret = false; char *group_sid_string; - struct dom_sid group_sid; + struct dom_sid *group_sid = NULL; int rc; + enum idmap_error_code err; TALLOC_CTX *tmp_ctx = talloc_stackframe(); filter = talloc_asprintf(tmp_ctx, @@ -820,17 +843,20 @@ static bool ldapsam_gid_to_sid(struct pdb_methods *methods, gid_t gid, goto done; } - if (!string_to_sid(&group_sid, group_sid_string)) { + err = sss_idmap_sid_to_smb_sid(priv->ipasam_privates->idmap_ctx, + group_sid_string, &group_sid); + if (err != IDMAP_SUCCESS) { DEBUG(3, ("Error calling sid_string_talloc for sid '%s'\n", group_sid_string)); goto done; } - sid_copy(sid, &group_sid); + sid_copy(sid, group_sid); ret = true; - done: +done: + talloc_free(group_sid); TALLOC_FREE(tmp_ctx); return ret; } @@ -897,6 +923,7 @@ struct ldap_search_state { const char **attrs; int attrsonly; void *pagedresults_cookie; + struct sss_idmap_ctx *idmap_ctx; LDAPMessage *entries, *current_entry; bool (*ldap2displayentry)(struct ldap_search_state *state, @@ -1066,7 +1093,9 @@ static bool ldapuser2displayentry(struct ldap_search_state *state, { char **vals; size_t converted_size; - struct dom_sid sid; + struct dom_sid *sid = NULL; + enum idmap_error_code err; + bool res; /* FIXME: SB try to figure out which flags to set instead of hardcode them */ result->acct_flags = 66048; @@ -1128,14 +1157,17 @@ static bool ldapuser2displayentry(struct ldap_search_state *state, return false; } - if (!string_to_sid(&sid, vals[0])) { + err = sss_idmap_sid_to_smb_sid(state->idmap_ctx, vals[0], &sid); + if (err != IDMAP_SUCCESS) { DEBUG(0, ("Could not convert %s to SID\n", vals[0])); ldap_value_free(vals); return false; } ldap_value_free(vals); - if (!sid_peek_check_rid(get_global_sam_sid(), &sid, &result->rid)) { + res = sid_peek_check_rid(get_global_sam_sid(), sid, &result->rid); + talloc_free(sid); + if (!res) { DEBUG(0, ("sid does not belong to our domain\n")); return false; } @@ -1170,6 +1202,7 @@ static bool ldapsam_search_users(struct pdb_methods *methods, state->attrsonly = 0; state->pagedresults_cookie = NULL; state->entries = NULL; + state->idmap_ctx = ldap_state->ipasam_privates->idmap_ctx; state->ldap2displayentry = ldapuser2displayentry; if ((state->filter == NULL) || (state->attrs == NULL)) { @@ -1191,8 +1224,9 @@ static bool ldapgroup2displayentry(struct ldap_search_state *state, { char **vals = NULL; size_t converted_size; - struct dom_sid sid; + struct dom_sid *sid = NULL; uint16_t group_type; + enum idmap_error_code err; result->account_name = ""; result->fullname = ""; @@ -1268,8 +1302,10 @@ static bool ldapgroup2displayentry(struct ldap_search_state *state, return false; } - if (!string_to_sid(&sid, vals[0])) { + err = sss_idmap_sid_to_smb_sid(state->idmap_ctx, vals[0], &sid); + if (err != IDMAP_SUCCESS) { DEBUG(0, ("Could not convert %s to SID\n", vals[0])); + ldap_value_free(vals); return false; } @@ -1279,9 +1315,10 @@ static bool ldapgroup2displayentry(struct ldap_search_state *state, case SID_NAME_DOM_GRP: case SID_NAME_ALIAS: - if (!sid_peek_check_rid(get_global_sam_sid(), &sid, &result->rid) - && !sid_peek_check_rid(&global_sid_Builtin, &sid, &result->rid)) + if (!sid_peek_check_rid(get_global_sam_sid(), sid, &result->rid) + && !sid_peek_check_rid(&global_sid_Builtin, sid, &result->rid)) { + talloc_free(sid); DEBUG(0, ("SID is not in our domain\n")); return false; } @@ -1289,8 +1326,10 @@ static bool ldapgroup2displayentry(struct ldap_search_state *state, default: DEBUG(0,("unknown group type: %d\n", group_type)); + talloc_free(sid); return false; } + talloc_free(sid); result->acct_flags = 0; @@ -1327,6 +1366,7 @@ static bool ldapsam_search_grouptype(struct pdb_methods *methods, state->pagedresults_cookie = NULL; state->entries = NULL; state->group_type = type; + state->idmap_ctx = ldap_state->ipasam_privates->idmap_ctx; state->ldap2displayentry = ldapgroup2displayentry; if ((state->filter == NULL) || (state->attrs == NULL)) { @@ -1848,6 +1888,8 @@ static bool fill_pdb_trusted_domain(TALLOC_CTX *mem_ctx, char *dummy; bool res; struct pdb_trusted_domain *td; + struct dom_sid *sid = NULL; + enum idmap_error_code err; if (entry == NULL) { return false; @@ -1867,11 +1909,14 @@ static bool fill_pdb_trusted_domain(TALLOC_CTX *mem_ctx, LDAP_ATTRIBUTE_TRUST_SID)); ZERO_STRUCT(td->security_identifier); } else { - res = string_to_sid(&td->security_identifier, dummy); + err = sss_idmap_sid_to_smb_sid(ldap_state->ipasam_privates->idmap_ctx, + dummy, &sid); TALLOC_FREE(dummy); - if (!res) { + if (err != IDMAP_SUCCESS) { return false; } + sid_copy(&td->security_identifier, sid); + talloc_free(sid); } if (!smbldap_talloc_single_blob(td, priv2ld(ldap_state), entry, @@ -3634,12 +3679,13 @@ static NTSTATUS pdb_init_ipasam(struct pdb_methods **pdb_method, NTSTATUS status; char *dn = NULL; char *domain_sid_string = NULL; - struct dom_sid ldap_domain_sid; + struct dom_sid *ldap_domain_sid = NULL; char *bind_dn = NULL; char *bind_secret = NULL; LDAPMessage *result = NULL; LDAPMessage *entry = NULL; + enum idmap_error_code err; status = make_pdb_method(pdb_method); if (!NT_STATUS_IS_OK(status)) { @@ -3776,15 +3822,27 @@ static NTSTATUS pdb_init_ipasam(struct pdb_methods **pdb_method, entry, LDAP_ATTRIBUTE_SID); + err = sss_idmap_init(idmap_talloc, ldap_state->ipasam_privates, + idmap_talloc_free, + &ldap_state->ipasam_privates->idmap_ctx); + if (err != IDMAP_SUCCESS) { + DEBUG(1, ("Failed to setup idmap context.\n")); + return NT_STATUS_UNSUCCESSFUL; + } + if (domain_sid_string) { - if (!string_to_sid(&ldap_domain_sid, domain_sid_string)) { + err = sss_idmap_sid_to_smb_sid(ldap_state->ipasam_privates->idmap_ctx, + domain_sid_string, + &ldap_domain_sid); + if (err != IDMAP_SUCCESS) { DEBUG(1, ("pdb_init_ldapsam: SID [%s] could not be " "read as a valid SID\n", domain_sid_string)); ldap_msgfree(result); TALLOC_FREE(domain_sid_string); return NT_STATUS_INVALID_PARAMETER; } - sid_copy(&ldap_state->domain_sid, &ldap_domain_sid); + sid_copy(&ldap_state->domain_sid, ldap_domain_sid); + talloc_free(ldap_domain_sid); talloc_free(domain_sid_string); status = save_sid_to_secret(ldap_state); @@ -3793,8 +3851,6 @@ static NTSTATUS pdb_init_ipasam(struct pdb_methods **pdb_method, } } - - (*pdb_method)->getsampwnam = ldapsam_getsampwnam; (*pdb_method)->search_users = ldapsam_search_users; (*pdb_method)->search_groups = ldapsam_search_groups; -- 1.7.11.4 -------------- next part -------------- From 2a612c1a31c5478d56aa407ae98569c7312e9e6c Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Tue, 28 Aug 2012 10:53:38 +0200 Subject: [PATCH 57/59] ipasam: replace get_global_sam_sid() --- daemons/ipa-sam/ipa_sam.c | 16 ++++++++++++---- 1 Datei ge?ndert, 12 Zeilen hinzugef?gt(+), 4 Zeilen entfernt(-) diff --git a/daemons/ipa-sam/ipa_sam.c b/daemons/ipa-sam/ipa_sam.c index 58a598f5aaf88dfeb95a76adef3a79b085638cd0..65e32c7dd542d3fbaed4e28a14d23a8e901d18c1 100644 --- a/daemons/ipa-sam/ipa_sam.c +++ b/daemons/ipa-sam/ipa_sam.c @@ -924,6 +924,7 @@ struct ldap_search_state { int attrsonly; void *pagedresults_cookie; struct sss_idmap_ctx *idmap_ctx; + const struct dom_sid *dom_sid; LDAPMessage *entries, *current_entry; bool (*ldap2displayentry)(struct ldap_search_state *state, @@ -1165,7 +1166,7 @@ static bool ldapuser2displayentry(struct ldap_search_state *state, } ldap_value_free(vals); - res = sid_peek_check_rid(get_global_sam_sid(), sid, &result->rid); + res = sid_peek_check_rid(state->dom_sid, sid, &result->rid); talloc_free(sid); if (!res) { DEBUG(0, ("sid does not belong to our domain\n")); @@ -1203,6 +1204,7 @@ static bool ldapsam_search_users(struct pdb_methods *methods, state->pagedresults_cookie = NULL; state->entries = NULL; state->idmap_ctx = ldap_state->ipasam_privates->idmap_ctx; + state->dom_sid = &ldap_state->domain_sid; state->ldap2displayentry = ldapuser2displayentry; if ((state->filter == NULL) || (state->attrs == NULL)) { @@ -1315,7 +1317,7 @@ static bool ldapgroup2displayentry(struct ldap_search_state *state, case SID_NAME_DOM_GRP: case SID_NAME_ALIAS: - if (!sid_peek_check_rid(get_global_sam_sid(), sid, &result->rid) + if (!sid_peek_check_rid(state->dom_sid, sid, &result->rid) && !sid_peek_check_rid(&global_sid_Builtin, sid, &result->rid)) { talloc_free(sid); @@ -1338,7 +1340,7 @@ static bool ldapgroup2displayentry(struct ldap_search_state *state, static bool ldapsam_search_grouptype(struct pdb_methods *methods, struct pdb_search *search, - const struct dom_sid *sid, + const struct dom_sid *sid, enum lsa_SidType type) { struct ldapsam_privates *ldap_state = @@ -1367,6 +1369,7 @@ static bool ldapsam_search_grouptype(struct pdb_methods *methods, state->entries = NULL; state->group_type = type; state->idmap_ctx = ldap_state->ipasam_privates->idmap_ctx; + state->dom_sid = &ldap_state->domain_sid; state->ldap2displayentry = ldapgroup2displayentry; if ((state->filter == NULL) || (state->attrs == NULL)) { @@ -1384,7 +1387,12 @@ static bool ldapsam_search_grouptype(struct pdb_methods *methods, static bool ldapsam_search_groups(struct pdb_methods *methods, struct pdb_search *search) { - return ldapsam_search_grouptype(methods, search, get_global_sam_sid(), SID_NAME_DOM_GRP); + struct ldapsam_privates *ldap_state = + (struct ldapsam_privates *)methods->private_data; + + return ldapsam_search_grouptype(methods, search, + &ldap_state->domain_sid, + SID_NAME_DOM_GRP); } static bool ldapsam_search_aliases(struct pdb_methods *methods, -- 1.7.11.4 -------------- next part -------------- From fdf0a4fe9e3bc682a612abfec510cd13682d4436 Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Tue, 28 Aug 2012 10:56:21 +0200 Subject: [PATCH 58/59] ipasam: remove fetch_ldap_pw() --- daemons/ipa-sam/ipa_sam.c | 12 ++---------- 1 Datei ge?ndert, 2 Zeilen hinzugef?gt(+), 10 Zeilen entfernt(-) diff --git a/daemons/ipa-sam/ipa_sam.c b/daemons/ipa-sam/ipa_sam.c index 65e32c7dd542d3fbaed4e28a14d23a8e901d18c1..95add09d15cb42b2603a038a977b817efb63b1d5 100644 --- a/daemons/ipa-sam/ipa_sam.c +++ b/daemons/ipa-sam/ipa_sam.c @@ -82,7 +82,6 @@ struct trustAuthInOutBlob { enum ndr_err_code ndr_pull_trustAuthInOutBlob(struct ndr_pull *ndr, int ndr_flags, struct trustAuthInOutBlob *r); /*available in libndr-samba.so */ -bool fetch_ldap_pw(char **dn, char** pw); /* available in libpdb.so */ bool sid_check_is_builtin(const struct dom_sid *sid); /* available in libpdb.so */ /* available in libpdb.so, renamed from sid_check_is_domain() in c43505b621725c9a754f0ee98318d451b093f2ed */ bool sid_linearize(char *outbuf, size_t len, const struct dom_sid *sid); /* available in libsmbconf.so */ @@ -3688,8 +3687,6 @@ static NTSTATUS pdb_init_ipasam(struct pdb_methods **pdb_method, char *dn = NULL; char *domain_sid_string = NULL; struct dom_sid *ldap_domain_sid = NULL; - char *bind_dn = NULL; - char *bind_secret = NULL; LDAPMessage *result = NULL; LDAPMessage *entry = NULL; @@ -3723,13 +3720,8 @@ static NTSTATUS pdb_init_ipasam(struct pdb_methods **pdb_method, status = ipasam_generate_principals(ldap_state->ipasam_privates); if (!NT_STATUS_IS_OK(status)) { - if (!fetch_ldap_pw(&bind_dn, &bind_secret)) { - DEBUG(0, ("pdb_init_ipasam: Failed to retrieve LDAP password from secrets.tdb\n")); - return NT_STATUS_NO_MEMORY; - } - status = smbldap_init(*pdb_method, pdb_get_tevent_context(), - uri, false, bind_dn, bind_secret, - &ldap_state->smbldap_state); + DEBUG(0, ("Failed to generate kerberos principal for LDAP authentication.\n")); + return status; } else { /* We authenticate via GSSAPI and thus will use kerberos principal to bind our access */ status = smbldap_init(*pdb_method, pdb_get_tevent_context(), -- 1.7.11.4 -------------- next part -------------- From 7312f3d01b2ca420845980bd4dc62cbdeac36108 Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Tue, 28 Aug 2012 13:03:13 +0200 Subject: [PATCH 59/59] ipasam: replace trim_char() with trim_string() --- daemons/ipa-sam/ipa_sam.c | 3 +-- 1 Datei ge?ndert, 1 Zeile hinzugef?gt(+), 2 Zeilen entfernt(-) diff --git a/daemons/ipa-sam/ipa_sam.c b/daemons/ipa-sam/ipa_sam.c index 95add09d15cb42b2603a038a977b817efb63b1d5..b3c336443d28d6850a283a373351043b2460eeaa 100644 --- a/daemons/ipa-sam/ipa_sam.c +++ b/daemons/ipa-sam/ipa_sam.c @@ -87,7 +87,6 @@ bool sid_check_is_builtin(const struct dom_sid *sid); /* available in libpdb.so bool sid_linearize(char *outbuf, size_t len, const struct dom_sid *sid); /* available in libsmbconf.so */ char *sid_string_talloc(TALLOC_CTX *mem_ctx, const struct dom_sid *sid); /* available in libsmbconf.so */ char *sid_string_dbg(const struct dom_sid *sid); /* available in libsmbconf.so */ -bool trim_char(char *s,char cfront,char cback); /* available in libutil_str.so */ char *escape_ldap_string(TALLOC_CTX *mem_ctx, const char *s); /* available in libsmbconf.so */ bool secrets_store(const char *key, const void *data, size_t size); /* available in libpdb.so */ @@ -3715,7 +3714,7 @@ static NTSTATUS pdb_init_ipasam(struct pdb_methods **pdb_method, if (uri == NULL) { return NT_STATUS_NO_MEMORY; } - trim_char( uri, '\"', '\"' ); + trim_string( uri, "\"", "\"" ); status = ipasam_generate_principals(ldap_state->ipasam_privates); -- 1.7.11.4 From pspacek at redhat.com Mon Sep 3 08:18:16 2012 From: pspacek at redhat.com (Petr Spacek) Date: Mon, 03 Sep 2012 10:18:16 +0200 Subject: [Freeipa-devel] Reevaluation of confirmation of actions in Web UI. In-Reply-To: <5040D179.2030206@redhat.com> References: <5040D179.2030206@redhat.com> Message-ID: <504467C8.8050408@redhat.com> On 08/31/2012 05:00 PM, Petr Vobornik wrote: > Hi Endi, > > I opened https://fedorahosted.org/freeipa/ticket/3035 can you please comment? > If everything seems good, I will try to implement some stuff to RC1 (next week). > > Thanks Hello, I'm not Endi, but I will add my bit: Usually I'm very frustrated from various confirmation dialogs. IMHO it should be configurable per-user. Each user is scared from different actions :-) Per-user setting will allow to enable confirmation only for "dangerous" operations, as a user feels it. -- Petr^2 Spacek From pviktori at redhat.com Mon Sep 3 10:30:17 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Mon, 03 Sep 2012 12:30:17 +0200 Subject: [Freeipa-devel] [PATCH] 0075 Add nsds5ReplicaStripAttrs to replica agreements Message-ID: <504486B9.6070403@redhat.com> Generalize the fix_replica_memberof update plugin to allow updating more replication attributes. Add nsds5ReplicaStripAttrs to replication agreements on update and replica install. https://fedorahosted.org/freeipa/ticket/2534 -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0075-Add-nsds5ReplicaStripAttrs-to-replica-agreements.patch Type: text/x-patch Size: 8005 bytes Desc: not available URL: From pvoborni at redhat.com Mon Sep 3 10:59:33 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Mon, 03 Sep 2012 12:59:33 +0200 Subject: [Freeipa-devel] [PATCH] 199 Permissions: select only applicable options on type change In-Reply-To: <503CD9AA.8090207@redhat.com> References: <5034CDD9.2050900@redhat.com> <503CD9AA.8090207@redhat.com> Message-ID: <50448D95.2080605@redhat.com> On 08/28/2012 04:46 PM, Endi Sukma Dewata wrote: > Found a couple of issues with Undo: Updated patch attached. > > 1. Using the scenario described in the ticket, if I undo the Type back > to User Group the Attributes aren't updated, it still shows the Service > attributes. Fixed. Now after undo it performs the same change as when changing the type. The fix is not very clean because it is dependent on the order of registration of undo_clicked event handlers, it should suffice. > > 2. After that, if I undo the Attributes it will show the originally > selected attribute (description) but the attribute will appear at the > end of Service attributes (not User Group attributes) and the attributes > are not sorted. Fixed by #1. When not performing #1 before, it still behaves this way. I think it is correct behavior otherwise it would still show the undo button. > > I also have some comments below. > > On 8/22/2012 7:17 AM, Petr Vobornik wrote: >> Problem: >> When a permission is edited, and Type switched, the attributes >> selected for previous Type are still selected, and update fails, if they >> are invalid for the new Type. But it should get deselected or not even >> listed if Type changes. >> >> Fix: >> When Type is changed, attribute list is refreshed and still applicable >> attributes are chosen. If Type is reverted back, previously chosen >> attributes are back as chosen. >> >> If attributes are extended outside Web UI by not listed attr, this >> attr is listed at the list end. > > To my understanding the list of ACI attributes are obtained from the > LDAP schema, so if a new attribute is added to the object class the UI > will know about it and show it in the attribute list. However, if the > attribute is added using the extensibleObject the UI may not know about > it because there's no schema change, is this what you meant? In that > case the UI won't show a checkbox for the attribute, so we'd probably > have to use the Filter or Subtree permission target that accepts > arbitrary attributes. On mod or create, validation of attrs is checking presence of attr in schema but not in object type's object classes. So in CLI admin can define attrs which are not offered in UI. It might be wrong, but Web UI should be able to work with it to avoid possible problems with dirty status right after load. > > Ideally the server should support a generic LDAP ACI target which would > accept any combination of LDAP filter, subtree, and attributes, but that > probably depends on the actual needs. > >> Note: >> If user makes change in attribute list before type change, this change >> is forgotten. >> >> https://fedorahosted.org/freeipa/ticket/2617 > -- Petr Vobornik -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pvoborni-0199-1-Permissions-select-only-applicable-options-on-type-c.patch Type: text/x-patch Size: 6184 bytes Desc: not available URL: From pvoborni at redhat.com Mon Sep 3 11:28:19 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Mon, 03 Sep 2012 13:28:19 +0200 Subject: [Freeipa-devel] [PATCH] 197 Fixed search in HBAC test In-Reply-To: <503CD8EC.6050503@redhat.com> References: <50339523.1070909@redhat.com> <503CD8EC.6050503@redhat.com> Message-ID: <50449453.2010404@redhat.com> On 08/28/2012 04:42 PM, Endi Sukma Dewata wrote: > On 8/21/2012 9:03 AM, Petr Vobornik wrote: >> Search in HBAC test wasn't working because expired flag wasn't set. >> >> https://fedorahosted.org/freeipa/ticket/2931 >> >> Notes: HBAC facets don't have refresh button. They can be refreshed by >> changing filter and searching. If one search with same filter, it sets >> expired flag but it doesn't refresh (search) because page state isn't >> changed. It refreshes when one go to different facet and returns back. >> Is this behavior acceptable? Or should we >> a) don't set expired flag when searching with unchanged filter >> b) force refresh when searching with unchanged filter >> c) add refresh button along with a) >> I prefer leave it as is or b) > > Hmm.. the behavior is consistent with the rest of the UI, but I think > when the user hits Enter or clicks the Search icon in the filter box > he'd expect the UI to run a new search and return the latest result even > if it's the same filter. I agree with option (b). A Refresh button can > be added too, but it's optional. > > So this patch is ACKed, but feel free to make a future improvement. > Maybe instead of having an 'expired flag' we could store an 'expiration > date'. If the user returns to the page before it expires, the UI can > show the old data. Otherwise the UI will rerun the search. > I did (b). Updated patch attached. I don't want to implement 'expiration date' at the moment. It's too widespread change. Maybe in FreeIPA 3.2. -- Petr Vobornik -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pvoborni-0197-1-Fixed-search-in-HBAC-test.patch Type: text/x-patch Size: 1333 bytes Desc: not available URL: From pviktori at redhat.com Mon Sep 3 11:53:41 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Mon, 03 Sep 2012 13:53:41 +0200 Subject: [Freeipa-devel] [PATCH 78] Ticket #2979 - prevent last admin from being disabled In-Reply-To: <503A5A8B.1050602@redhat.com> References: <502D9291.2090102@redhat.com> <503275D9.2020907@redhat.com> <503A5A8B.1050602@redhat.com> Message-ID: <50449A45.3090608@redhat.com> On 08/26/2012 07:19 PM, John Dennis wrote: > On 08/20/2012 01:37 PM, Petr Viktorin wrote: >> (Sorry if you're getting this twice; I didn't send it to the list) >> >> On 08/16/2012 08:38 PM, John Dennis wrote: >>> >>> -- >>> John Dennis >>> >>> Looking to carve out IT costs? >>> www.redhat.com/carveoutcosts/ >>> >>> freeipa-jdennis-0078-Ticket-2979-prevent-last-admin-from-being-disabled.patch >>> >>> >>> >>> >From c47109c63530e188db76986fdda48c76bf681d10 Mon Sep 17 00:00:00 2001 >>> From: John Dennis >>> Date: Thu, 16 Aug 2012 20:28:44 -0400 >>> Subject: [PATCH 78] Ticket #2979 - prevent last admin from being >>> disabled >>> Content-Type: text/plain; charset="utf-8" >>> Content-Transfer-Encoding: 8bit >>> >>> We prevent the last member of the admin group from being deleted. The >>> same check needs to be performed when disabling a user. >>> >>> Moved the code in del_user to a common subroutine and call it from >>> both user_del and user_disable. Note, unlike user_del user_disable >>> does not have a 'pre' callback therefore the check function is called >>> in user_disable's execute routine. >> >> This should also prevent disabling all admins if there's more than one: >> >> # ipa user-add admin2 --first=a --last=b >> ------------------- >> Added user "admin2" >> ------------------- >> ... >> # ipa group-add-member admins --user=admin2 >> ------------------------- >> Number of members added 1 >> ------------------------- >> # ipa user-disable admin2 >> ------------------------------ >> Disabled user account "admin2" >> ------------------------------ >> # ipa user-disable admin >> ------------------------------ >> Disabled user account "admin" >> ------------------------------ >> # ipa ping >> ipa: ERROR: Server is unwilling to perform: Account inactivated. Contact >> system administrator. >> >> Also with one enabled and one disabled admin, it shouldn't be possible >> to delete the enabled one. >> >> >> Please add some tests; you can extend the ones added in commit f8e7b51. > > Good catch with respect to disabled users, thank you. > > Reworked patch attached, see patch comments. > > > > Works well now, just the error message is incorrect: it mentions only deleting, not disabling. $ ipa user-disable admin ipa: ERROR: admin cannot be deleted because it is the last member of group admins -- Petr? From abokovoy at redhat.com Mon Sep 3 12:03:18 2012 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Mon, 3 Sep 2012 08:03:18 -0400 (EDT) Subject: [Freeipa-devel] [PATCHES] Remove dependencies to private samba libs In-Reply-To: <20120902170422.GC2071@localhost.localdomain> Message-ID: <1644106786.2826837.1346673798761.JavaMail.root@redhat.com> ----- Original Message ----- > From: "Sumit Bose" > To: "freeipa-devel" > Sent: Sunday, September 2, 2012 8:04:22 PM > Subject: [Freeipa-devel] [PATCHES] Remove dependencies to private samba libs > > Hi, > > private samba libraries will become a new version with every new > samba > release. To avoid rebuilding of IPA whenever a new samba version is > released the following series of patches removes the dependencies to > private samba libraries by replacing the related calls with others. > > This should fix https://fedorahosted.org/freeipa/ticket/3013 ACK to all of them, I checked and they work correctly. I had concerns about strdup upper patch but talloc_strdup_upper() avoids reallocation of memory for our use case (storing security key). -- / Alexander Bokovoy From mkosek at redhat.com Mon Sep 3 12:35:23 2012 From: mkosek at redhat.com (Martin Kosek) Date: Mon, 03 Sep 2012 14:35:23 +0200 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <5040CFEF.8040100@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502B86F5.4070008@redhat.com> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> Message-ID: <5044A40B.6070604@redhat.com> On 08/31/2012 04:53 PM, Petr Viktorin wrote: > On 08/28/2012 03:40 PM, Petr Viktorin wrote: >> On 08/17/2012 06:04 PM, Ade Lee wrote: >>> On Fri, 2012-08-17 at 09:34 -0400, Ade Lee wrote: >>>> On Thu, 2012-08-16 at 18:45 +0200, Martin Kosek wrote: >>>>> On 08/16/2012 01:28 PM, Ade Lee wrote: >>>>>> Patch attached this time. I should know better than to do this in the >>>>>> middle of the night .. >>>>>> >>>>>> On Thu, 2012-08-16 at 09:12 +0200, Martin Kosek wrote: >>>>>>> On 08/16/2012 07:53 AM, Ade Lee wrote: >>>>>>>> On Wed, 2012-08-15 at 23:41 -0400, Ade Lee wrote: >>>>>>>>> On Wed, 2012-08-15 at 16:34 +0200, Martin Kosek wrote: >>>>>>>>>> On 08/15/2012 03:54 PM, Ade Lee wrote: >>>>>>>>>>> On Wed, 2012-08-15 at 13:24 +0200, Martin Kosek wrote: >>>>>>>>>>>> On 08/08/2012 10:05 PM, Ade Lee wrote: >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> Dogtag 10 is being released on f18, and has a number of >>>>>>>>>>>>> changes that >>>>>>>>>>>>> will affect IPA. In particular, the following changes will >>>>>>>>>>>>> affect >>>>>>>>>>>>> current IPA code. >>>>>>>>>>>>> >>>>>>>>>>>>> * The directory layout of the dogtag instance has changed. >>>>>>>>>>>>> Instead of >>>>>>>>>>>>> using separate tomcat instances to host different >>>>>>>>>>>>> subsystems, the >>>>>>>>>>>>> standard dogtag installation will allow one to install a CA. >>>>>>>>>>>>> KRA, OCSP >>>>>>>>>>>>> and TKS within the same instance. There have been >>>>>>>>>>>>> corresponding changes >>>>>>>>>>>>> in the directory layout, as well as the default instance name >>>>>>>>>>>>> (pki-tomcat instead of pki-ca), and startup daemon >>>>>>>>>>>>> (pki-tomcatd, instead >>>>>>>>>>>>> of pki-cad, pki-krad etc.) >>>>>>>>>>>>> >>>>>>>>>>>>> * The default instance will use only four ports (HTTPS, >>>>>>>>>>>>> HTTP, AJP and >>>>>>>>>>>>> tomcat shutdown port) rather than the 6 previously used. >>>>>>>>>>>>> The default >>>>>>>>>>>>> ports will be changed to the standard tomcat ports. As >>>>>>>>>>>>> these ports are >>>>>>>>>>>>> local to the ipa server machine, this should not cause too much >>>>>>>>>>>>> disruption. >>>>>>>>>>>>> >>>>>>>>>>>>> * There is a new single step installer written in python. >>>>>>>>>>>>> (pkispawn/destroy) vs. pkicreate/pkisilent/pkiremove. >>>>>>>>>>>>> >>>>>>>>>>>>> * Dogtag 10 runs on tomcat7 - with a new corresponding >>>>>>>>>>>>> version of >>>>>>>>>>>>> tomcatjss. >>>>>>>>>>>>> >>>>>>>>>>>>> The attached patch integrates all the above changes in IPA >>>>>>>>>>>>> installation >>>>>>>>>>>>> and maintenance code. Once the patch is applied, users will >>>>>>>>>>>>> be able to: >>>>>>>>>>>>> >>>>>>>>>>>>> 1. run ipa-server-install to completion on f18 with dogtag 10. >>>>>>>>>>>>> 2. install a new replica on f18 on dogtag 10. >>>>>>>>>>>>> 3. upgrade an f17 machine with an existing IPA instance to >>>>>>>>>>>>> f18/ dogtag >>>>>>>>>>>>> 10 - and have that old-style dogtag instance continue to run >>>>>>>>>>>>> correctly. >>>>>>>>>>>>> This will require the installation of the latest version of >>>>>>>>>>>>> tomcatjss as >>>>>>>>>>>>> well as the installation of tomcat6. The old-style instance >>>>>>>>>>>>> will >>>>>>>>>>>>> continue to use tomcat6. >>>>>>>>>>>>> 4. in addition, the new cert renewal code has been patched >>>>>>>>>>>>> and should >>>>>>>>>>>>> continue to work. >>>>>>>>>>>>> >>>>>>>>>>>>> What is not yet completed / supported: >>>>>>>>>>>>> >>>>>>>>>>>>> 1. Installation with an external CA is not yet completed in >>>>>>>>>>>>> the new >>>>>>>>>>>>> installer. We plan to complete this soon. >>>>>>>>>>>>> >>>>>>>>>>>>> 2. There is some IPA upgrade code that has not yet been touched >>>>>>>>>>>>> (install/tools/ipa-upgradeconfig). >>>>>>>>>>>>> >>>>>>>>>>>>> 3. A script needs to be written to allow admins to convert >>>>>>>>>>>>> their >>>>>>>>>>>>> old-style dogtag instances to new style instances, as well >>>>>>>>>>>>> as code to >>>>>>>>>>>>> periodically prompt admins to do this. >>>>>>>>>>>>> >>>>>>>>>>>>> 4. Installation of old-style instances using >>>>>>>>>>>>> pkicreate/pkisilent on >>>>>>>>>>>>> dogtag 10 will no longer be supported, and will be disabled >>>>>>>>>>>>> soon. >>>>>>>>>>>>> >>>>>>>>>>>>> 5. The pki-selinux policy has been updated to reflect these >>>>>>>>>>>>> changes, >>>>>>>>>>>>> but is still in flux. In fact, it is our intention to place >>>>>>>>>>>>> the dogtag >>>>>>>>>>>>> selinux policy in the base selinux policy for f18. In the >>>>>>>>>>>>> meantime, it >>>>>>>>>>>>> may be necessary to run installs in permissive mode. >>>>>>>>>>>>> >>>>>>>>>>>>> The dogtag 10 code will be released shortly into f18. Prior >>>>>>>>>>>>> to that >>>>>>>>>>>>> though, we have placed the new dogtag 10 and tomcatjss code >>>>>>>>>>>>> in a >>>>>>>>>>>>> developer repo that is located at >>>>>>>>>>>>> http://nkinder.fedorapeople.org/dogtag-devel/ >>>>>>>>>>>>> >>>>>>>>>>>>> Testing can be done on both f18 and f17 - although the >>>>>>>>>>>>> target platform - >>>>>>>>>>>>> and the only platform for which official builds will be >>>>>>>>>>>>> created is f18. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Ade >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hi Ade, >>>>>>>>>>>> >>>>>>>>>>>> Thanks for the patch, I started with review and integration >>>>>>>>>>>> tests (currently >>>>>>>>>>>> running on Fedora 17 with Nathan's repo). >>>>>>>>>>>> >>>>>>>>>>>> Installation on single master was smooth, it worked just >>>>>>>>>>>> fine, even with >>>>>>>>>>>> enforced SELinux, without any error - kudos to you and the >>>>>>>>>>>> whole dogtag team. >>>>>>>>>>>> The resulting logs and the structure of your log directory >>>>>>>>>>>> seems improved. I >>>>>>>>>>>> believe that the brand new Python installers will make it >>>>>>>>>>>> easier to debug >>>>>>>>>>>> issues with dogtag installation when they come. When I tried >>>>>>>>>>>> our unit tests or >>>>>>>>>>>> some simple cert operation, it worked fine as well. >>>>>>>>>>>> >>>>>>>>>>>> Now the bad news, or rather few issues and suggestions I found: >>>>>>>>>>>> >>>>>>>>>>>> 1) As we already discussed on IRC, tomcat 7 was not pulled >>>>>>>>>>>> automatically on >>>>>>>>>>>> Fedora 17 when I updated pki-ca, you somewhere miss a Requires. >>>>>>>>>>>> >>>>>>>>>>> We have a dogtag patch that is currently in review that will >>>>>>>>>>> address >>>>>>>>>>> this. Once this is in, tomcatjss >=7.0.0 will be required for >>>>>>>>>>> f17+, >>>>>>>>>>> rather than f18+ >>>>>>>>>>> >>>>>>>>>>>> 2) I had installed IPA with dogtag10 on master. However, CA >>>>>>>>>>>> installation on a >>>>>>>>>>>> replica (ipa-ca-install) with dogtag9 failed - is this >>>>>>>>>>>> expectable? >>>>>>>>>>>> >>>>>>>>>>> Yes. The current IPA patch is designed to work with dogtag 10 >>>>>>>>>>> only, >>>>>>>>>>> which will be officially available on f18+. So if you update >>>>>>>>>>> to dogtag >>>>>>>>>>> 10, you must have this patch and visa versa. We probably need >>>>>>>>>>> to add >>>>>>>>>>> the relevant requires to IPA to enforce this. >>>>>>>>>>> >>>>>>>>>>> If you have an existing dogtag 9 instance, and you upgrade to >>>>>>>>>>> the new >>>>>>>>>>> dogtag 10 and patched IPA, then that instance will continue to >>>>>>>>>>> work. >>>>>>>>>>> But any new instances would be created using dogtag 10. >>>>>>>>>>> >>>>>>>>>>>> 3) I had installed IPA with dogtag10 on master. Replica had >>>>>>>>>>>> dogtag10 as well >>>>>>>>>>>> and I got the following error: >>>>>>>>>>>> >>>>>>>>>>>> # ipa-ca-install >>>>>>>>>>>> /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg >>>>>>>>>>>> ... >>>>>>>>>>>> Configuring certificate server: Estimated time 3 minutes 30 >>>>>>>>>>>> seconds >>>>>>>>>>>> [1/14]: creating certificate server user >>>>>>>>>>>> [2/14]: configuring certificate server instance >>>>>>>>>>>> >>>>>>>>>>>> Your system may be partly configured. >>>>>>>>>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up. >>>>>>>>>>>> >>>>>>>>>>>> Unexpected error - see /var/log/ipareplica-ca-install.log for >>>>>>>>>>>> details: >>>>>>>>>>>> IOError: [Errno 2] No such file or directory: >>>>>>>>>>>> '/var/lib/pki/pki-tomcat/alias/ca_backup_keys.p12' >>>>>>>>>>>> >>>>>>>>>>>> Root cause: >>>>>>>>>>>> ... >>>>>>>>>>>> File >>>>>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", >>>>>>>>>>>> line >>>>>>>>>>>> 625, in __spawn_instance >>>>>>>>>>>> "/root/cacert.p12") >>>>>>>>>>>> ... >>>>>>>>>>>> >>>>>>>>>>> I need to look into this. I had fixed ipa-replica-install, >>>>>>>>>>> rather than >>>>>>>>>>> ipa-ca-install to create replicas. I didn't know ipa-ca-install >>>>>>>>>>> existed! Should not be too bad to fix though - most likely >>>>>>>>>>> just need to >>>>>>>>>>> move files to the right place. >>>>>>>>>> >>>>>>>>>> Ok, thanks! Btw. CA on replica can be either installed during >>>>>>>>>> ipa-replica-install time (when --setup-ca option is passed, you >>>>>>>>>> probably used >>>>>>>>>> that one) and the aforementioned ipa-ca-install run after >>>>>>>>>> ipa-replica-install. >>>>>>>>>> >>>>>>>>> I will be testing this out again. As ipa-ca-install uses the >>>>>>>>> same code >>>>>>>>> as ipa-replica-install, I would have expected it to work. Did >>>>>>>>> you try >>>>>>>>> it with selinux in permissive mode? >>>>>>>>> >>>>>>>> OK -- I tested this out and realized that between the time I >>>>>>>> initially >>>>>>>> tested this and your tests, we had changed a URL >>>>>>>> from /ca/pki/installer/installToken to >>>>>>>> /ca/rest/installer/installToken, >>>>>>>> but I forgot to update ipa-pki-proxy.conf. >>>>>>>> >>>>>>>> The new patch has been rebased and changes the relevant patch. With >>>>>>>> this, the CA replica installs correctly. >>>>>>> >>>>>>> Umh, where can I find the new rebased patch? :-) >>>>>>> >>>>>>>> >>>>>>>> There was one issue that I encountered. I have seen this once >>>>>>>> before >>>>>>>> and will need to investigate further. IPA sometimes restarts the >>>>>>>> dogtag >>>>>>>> instance by doing systemctl stop pki-tomcatd.target. and then >>>>>>>> systemctl >>>>>>>> start pki-tomcatd.target. I have noticed that occasionally the >>>>>>>> start >>>>>>>> invocation fails, while the stop and restart invocations succeed >>>>>>>> just >>>>>>>> fine. This makes absolutely no sense because restart is essentially >>>>>>>> just stop and then start. >>>>>>>> >>>>>>>> I think this is actually a bug in systemd. If I reboot the >>>>>>>> machine, it >>>>>>>> all works perfectly. If we can't figure out whats going on with >>>>>>>> systemd, we can always replace this start/stop with >>>>>>>> starting/stopping >>>>>>>> the service directly. (pki-tomcatd at pki-tomcat.service). That >>>>>>>> seems to >>>>>>>> work all the time with no problems. >>>>>>>> >>>>>>>> I suggest we leave this fix (if needed) for a separate patch. >>>>>>> >>>>>>> Ok, I will see if I hit this issue again. Btw. is it recommended >>>>>>> in systemd to >>>>>>> use target units this way? I thought that only service files are >>>>>>> meant to be >>>>>>> used for manipulation with a service. As you said yourself, using >>>>>>> service unit >>>>>>> file for restart worked every time. >>>>>>> >>>>>>> Martin >>>>>> >>>>> >>>>> Now, I tested the rebased patch with VMs with permissive SELinux. >>>>> IPA server >>>>> install was OK, as always, but replica installation ended up with >>>>> error. >>>>> Although it got much further than before: >>>>> >>>>> # ipa-ca-install >>>>> /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg >>>>> ... >>>>> [3/3]: restarting directory server >>>>> done configuring pkids. >>>>> Configuring certificate server: Estimated time 3 minutes 30 seconds >>>>> [1/14]: creating certificate server user >>>>> [2/14]: configuring certificate server instance >>>>> [3/14]: disabling nonces >>>>> [4/14]: importing CA chain to RA certificate database >>>>> [5/14]: fixing RA database permissions >>>>> [6/14]: setting up signing cert profile >>>>> [7/14]: set up CRL publishing >>>>> [8/14]: set certificate subject base >>>>> [9/14]: enabling Subject Key Identifier >>>>> [10/14]: configuring certificate server to start on boot >>>>> [11/14]: configure certmonger for renewals >>>>> [12/14]: configure clone certificate renewals >>>>> [13/14]: configure Server-Cert certificate renewal >>>>> [14/14]: Configure HTTP to proxy connections >>>>> done configuring pki-tomcatd. >>>>> Restarting the directory and certificate servers >>>>> >>>>> Your system may be partly configured. >>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up. >>>>> >>>>> It seems like that CA on a replica did not start and so a check for >>>>> port 8080 >>>>> failed. Attached logs (pki-ca-logs.tgz) may provide more information >>>>> to this issue. >>>>> >>>> This is the systemd issue I mentioned before. I will submit a patch >>>> which will change the restart mechanism to restart the service and not >>>> the target. >>>>> >>> >>> Patch attached. This patch should be applied on top of the large patch >>> (being reviewed). >>> >>>>> Now I am also testing upgrade from IPA+dogtag9 to PatchedIPA+dogtag10 >>>>> (permissive SELinux). Now, the service was upgraded smoothly, CA >>>>> service >>>>> could be restarted without failure. However, certificate operations now >>>>> did not work: >>>>> >>>>> # ipactl restart >>>>> Restarting Directory Service >>>>> Restarting KDC Service >>>>> Restarting KPASSWD Service >>>>> Restarting MEMCACHE Service >>>>> Restarting HTTP Service >>>>> Restarting CA Service >>>>> # ipactl status >>>>> Directory Service: RUNNING >>>>> KDC Service: RUNNING >>>>> KPASSWD Service: RUNNING >>>>> MEMCACHE Service: RUNNING >>>>> HTTP Service: RUNNING >>>>> CA Service: RUNNING >>>>> # ipa cert-show 1 >>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to >>>>> communicate with CMS (Internal Server Error) >>>>> >>>>> Attaching logs for this case as well (ipa-ca-logs-upgrade.tgz). >>>>> >>>> The reason for this is that the old dogtag instances are missing: >>>> 1. a new parameter in CS.cfg >>>> 2. a couple of symlinks >>>> >>>> I plan to fix this in the dogtag code. Specifically, >>>> 1. I will modify the dogtag code to provide a default value in the case >>>> where the parameter does not exist. >>>> 2. I plan to add a function to the dogtag startup scripts to check and >>>> remake any required symlinks. >>>> >>>> Both of these changes are in the dogtag code, and do not affect this >>>> patch. As a workaround, to confirm that the upgrade actually works, do >>>> the following manual steps on your upgraded instance: >>>> >>>> 1. Add the following parameter to CS.cfg >>>> pidDir=/var/run >>>> >>>> 2. Add the following links; >>>> cd /var/lib/pki-ca/common/lib >>>> ln -s /usr/share/java/apache-commons-codec.jar >>>> apache-commons-codec.jar >>>> ln -s /usr/share/java/log4j.jar log4j.jar >>>> >>>> 3 Restart the dogtag instance >>>> >>>>> HTH, >>>>> Martin >>>> >>>> >> >> The attached patch conditionalizes the changes, so that IPA supports >> both Dogtag versions. >> >> I didn't put it in platform code for two reasons >> - One platform (f17) can have either Dogtag version installed >> - I didn't want to conflict with Timo Aaltonen's "platformizing" work. >> According to his WIP patches, he plans to move the whole dogtag module >> into platform code. >> >> I used the existence of /usr/bin/pkispawn as a test for Dogtag 10. If >> you know a better way, please comment. >> >> The dogtag_version option is now always added to >> >> I've reverted the changes to install/ui/test/data/ipa_init.json, so >> you'll have to change the ports manually when testing the UI with Dogtag >> 10. >> >> >> > > Attaching a patch with fixed ipa-pki-proxy.conf, as discussed on IRC. > This approach looks good. I did not hit any regression on F17 with dogtag9, or clean installs of IPA+dogtag10. I think we are getting close to pushing this code. The only issues I hit so far are for upgraded dogtag9 instances (on F17): 1) The service did not start for me. There were some SELinux AVCs and even when I disabled SELinux the instance still did not start (logs attached). 2) Uninstall was also not clean, we leave some dogtag installation states for upgraded dogtag9 instance: # ipa-server-install --uninstall --unattended Shutting down all IPA services Removing IPA client configuration Unconfiguring ntpd Unconfiguring CA directory server Unconfiguring web server Unconfiguring krb5kdc Unconfiguring kadmin Unconfiguring directory server Unconfiguring ipa_memcached ipa : ERROR Some installation state for pki-cad has not been restored, see /var/lib/ipa/sysrestore/sysrestore.state /var/lib/ipa/sysrestore/sysrestore.state: [pki-cad] enabled = False Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: pki-ca-logs-upgrade.tgz Type: application/x-compressed-tar Size: 140426 bytes Desc: not available URL: From pvoborni at redhat.com Mon Sep 3 13:35:41 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Mon, 03 Sep 2012 15:35:41 +0200 Subject: [Freeipa-devel] [PATCH] 203 Notify success on add, delete and update Message-ID: <5044B22D.7030602@redhat.com> Notification of success was added to: * details facet: update * association facet and association widget: add, delete items * attribute facet: delete items (notification of add should be handled in entity adder dialog) * sudo rule: add, remove option * dnsrecord: add, update, delete https://fedorahosted.org/freeipa/ticket/2977 -- Petr Vobornik -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pvoborni-0203-Notify-success-on-add-delete-and-update.patch Type: text/x-patch Size: 15623 bytes Desc: not available URL: From jdennis at redhat.com Mon Sep 3 14:41:16 2012 From: jdennis at redhat.com (John Dennis) Date: Mon, 03 Sep 2012 10:41:16 -0400 Subject: [Freeipa-devel] [PATCH 78] Ticket #2979 - prevent last admin from being disabled In-Reply-To: <50449A45.3090608@redhat.com> References: <502D9291.2090102@redhat.com> <503275D9.2020907@redhat.com> <503A5A8B.1050602@redhat.com> <50449A45.3090608@redhat.com> Message-ID: <5044C18C.5060900@redhat.com> On 09/03/2012 07:53 AM, Petr Viktorin wrote: > On 08/26/2012 07:19 PM, John Dennis wrote: >> On 08/20/2012 01:37 PM, Petr Viktorin wrote: >>> (Sorry if you're getting this twice; I didn't send it to the list) >>> >>> On 08/16/2012 08:38 PM, John Dennis wrote: >>>> >>>> -- >>>> John Dennis >>>> >>>> Looking to carve out IT costs? >>>> www.redhat.com/carveoutcosts/ >>>> >>>> freeipa-jdennis-0078-Ticket-2979-prevent-last-admin-from-being-disabled.patch >>>> >>>> >>>> >>>> >From c47109c63530e188db76986fdda48c76bf681d10 Mon Sep 17 00:00:00 2001 >>>> From: John Dennis >>>> Date: Thu, 16 Aug 2012 20:28:44 -0400 >>>> Subject: [PATCH 78] Ticket #2979 - prevent last admin from being >>>> disabled >>>> Content-Type: text/plain; charset="utf-8" >>>> Content-Transfer-Encoding: 8bit >>>> >>>> We prevent the last member of the admin group from being deleted. The >>>> same check needs to be performed when disabling a user. >>>> >>>> Moved the code in del_user to a common subroutine and call it from >>>> both user_del and user_disable. Note, unlike user_del user_disable >>>> does not have a 'pre' callback therefore the check function is called >>>> in user_disable's execute routine. >>> >>> This should also prevent disabling all admins if there's more than one: >>> >>> # ipa user-add admin2 --first=a --last=b >>> ------------------- >>> Added user "admin2" >>> ------------------- >>> ... >>> # ipa group-add-member admins --user=admin2 >>> ------------------------- >>> Number of members added 1 >>> ------------------------- >>> # ipa user-disable admin2 >>> ------------------------------ >>> Disabled user account "admin2" >>> ------------------------------ >>> # ipa user-disable admin >>> ------------------------------ >>> Disabled user account "admin" >>> ------------------------------ >>> # ipa ping >>> ipa: ERROR: Server is unwilling to perform: Account inactivated. Contact >>> system administrator. >>> >>> Also with one enabled and one disabled admin, it shouldn't be possible >>> to delete the enabled one. >>> >>> >>> Please add some tests; you can extend the ones added in commit f8e7b51. >> >> Good catch with respect to disabled users, thank you. >> >> Reworked patch attached, see patch comments. >> >> >> >> > > Works well now, just the error message is incorrect: it mentions only > deleting, not disabling. > > $ ipa user-disable admin > ipa: ERROR: admin cannot be deleted because it is the last member of > group admins Updated the error message to say "... cannot be deleted or disabled because ..." -- John Dennis Looking to carve out IT costs? www.redhat.com/carveoutcosts/ -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-jdennis-0078-2-prevent-last-admin-from-being-disabled.patch Type: text/x-patch Size: 51035 bytes Desc: not available URL: From mkosek at redhat.com Mon Sep 3 14:42:56 2012 From: mkosek at redhat.com (Martin Kosek) Date: Mon, 03 Sep 2012 16:42:56 +0200 Subject: [Freeipa-devel] Ticket #2866 - referential integrity in IPA In-Reply-To: <503BA065.1060207@redhat.com> References: <1345211089.3796.34.camel@priserak> <502E4E78.1090407@redhat.com> <503B9F4E.4010005@redhat.com> <503BA065.1060207@redhat.com> Message-ID: <5044C1F0.7080609@redhat.com> On 08/27/2012 06:29 PM, Rich Megginson wrote: > On 08/27/2012 10:24 AM, Martin Kosek wrote: >> On 08/17/2012 04:00 PM, Rich Megginson wrote: >>> On 08/17/2012 07:44 AM, Martin Kosek wrote: >>>> Hi guys, >>>> >>>> I am now investigating ticket #2866: >>>> https://fedorahosted.org/freeipa/ticket/2866 >>>> >>>> And I am thinking about possible solutions for this problem. In a >>>> nutshell, we do not properly check referential integrity in some IPA >>>> objects where we keep one-way DN references to other objects, e.g. in >>>> - managedBy attribute for a host object >>>> - memberhost attribute for HBAC rule object >>>> - memberuser attribute for user object >>>> - memberallowcmd or memberdenycmd for SUDO command object (reported in >>>> #2866) >>>> ... >>>> >>>> Currently, I see 2 approaches to solve this: >>>> 1) Add relevant checks to our ipalib plugins where problematic >>>> operations with these operations are being executed (like we do for >>>> selinuxusermap's seealso attribute in HBAC plugin) >>>> This of course would not prevent direct LDAP deletes. >>>> >>>> 2) Implement a preop DS plugin that would hook to MODRDN and DELETE >>>> callbacks and check that this object's DN is not referenced in other >>>> objects. And if it does, it would reject such modification. Second >>>> option would be to delete the attribute value with now invalid >>>> reference. This would be probably more suitable for example for >>>> references to user objects. >>>> >>>> Any comments to these possible approaches are welcome. >>>> >>>> Rich, do you think that as an alternative to these 2 approaches, >>>> memberOf plugin could be eventually modified to do this task? >>> This is very similar to the referential integrity plugin already in 389, except >>> instead of cleaning up references to moved and deleted entries, you want it to >>> prevent moving or deleting an entry if that entry is referenced by the >>> managedby/memberhost/memberuser/memberallowcmd/memberdenycmd of some other >>> entry. >> I think that using or enhancing current DS referential integrity plugin will be >> the best and the most consistent way to go. >> >> We already use that plugin for some user attributes like "manager" or >> "secretary". seeAlso is already covered by default, so for example seeAlso >> attribute in SELinux usermap object referencing an HBAC rule will get removed >> when relevant HBAC rule is removed (I just checked that). >> >>> Note that the managed entry plugin (mep) already handles this for the managedby >>> attribute. >> I assume you are referencing "mepmanagedby" and "mepmanagedentry" attributes >> which then produce errors like this one: >> >> # ipa netgroup-del foo >> ipa: ERROR: Server is unwilling to perform: Deleting a managed entry is not >> allowed. It needs to be manually unlinked first. >> >> managedBy attribute used by hosts objects I had in mind seems to not be covered. >> >> But you are right, this is pretty much what I wanted. Though in case of MEP, >> there is a link in both referenced objects, but in our case, we have just the >> one-way link. >> >>> Are you already using the memberof plugin for >>> memberhost/memberuser/memberallowcmd/memberdenycmd? >>> >>> This doesn't seem like a job for memberof, this seems like more of a new check >>> for the referential integrity plugin. >>> >> I am now considering if current move/delete clean up already present in >> Referential Integrity plugin would not be sufficient for us. >> >> Rich, please correct me if I am wrong, but in that case, we would just need to >> add relevant attribute names >> (memberhost/memberuser/memberallowcmd/memberdenycmd...) to Referential >> Integrity plugin configuration as nsslapd-pluginarg7, nsslapd-pluginarg8, ... >> I wonder if there would be some performance issues if we add attributes to the >> list this way. > No, not if they are indexed for presence and equality. > Hello Rich, I am back to investigate this ticket. In order to be able to deliver some working solution to IPA 3.0, I plan to take advantage of current Referential Integrity Plugin to clean up dangling references. This is the plan I plan to take: 1) Add pres,eq indexes for all un-indexed attributes that we want to check: sourcehost memberservice managedby memberallowcmd memberdenycmd ipasudorunas ipasudorunasgroup 2) Add missing "pres" index for attributes we want to check, but only have eq index: manager secretary memberuser memberhost I assume this step is also needed in order to keep the server performance. 3) Add all these attributes do Referential Integrity Plugin attribute list of not already 4) Also add Index task (nsIndexAttribute) for all these new indexes so that they are created during IPA server upgrade. Is this procedure OK DS-wise? I also have question regarding the following note in RHDS doc chapter 3.6. Maintaining Referential Integrity: "The Referential Integrity Plug-in should only be enabled on one supplier replica in a multi-master replication environment to avoid conflict resolution loops..." Currently, we enable this plugin on all IPA replicas. Is this something we need to be concerned about and fix ASAP (before we do all this RefInt effort)? Thank you, Martin From tbabej at redhat.com Mon Sep 3 15:12:14 2012 From: tbabej at redhat.com (Tomas Babej) Date: Mon, 03 Sep 2012 17:12:14 +0200 Subject: [Freeipa-devel] [PATCH 0011] Make sure selinuxusemap behaves consistently to HBAC rule Message-ID: <5044C8CE.20108@redhat.com> Hi, Both selinuxusermap-add and selinuxusermap-mod commands now behave consistently in not allowing user/host category or user/host members and HBAC rule being set at the same time. Also adds a bunch of unit tests that check this behaviour. https://fedorahosted.org/freeipa/ticket/2983 Tomas -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-tbabej-0011-Make-sure-selinuxusemap-behaves-consistently-to-HBAC.patch Type: text/x-patch Size: 11338 bytes Desc: not available URL: From pviktori at redhat.com Mon Sep 3 15:14:12 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Mon, 03 Sep 2012 17:14:12 +0200 Subject: [Freeipa-devel] [PATCH] 0072 Internationalization for public errors In-Reply-To: <503E1D04.8070303@redhat.com> References: <500FFB88.6080902@redhat.com> <50338DB9.3050606@redhat.com> <503E1D04.8070303@redhat.com> Message-ID: <5044C944.1050509@redhat.com> On 08/29/2012 03:45 PM, Martin Kosek wrote: > On 08/21/2012 03:31 PM, Petr Viktorin wrote: >> On 07/25/2012 09:58 AM, Petr Viktorin wrote: >>> I spent some time while IPA was installing tracking down error mesages >>> that weren't marked for translation :) >>> I also changed the error docstrings, as people are likely to look there >>> for inspiration. >>> >>> >>> https://fedorahosted.org/freeipa/ticket/1953 >> >> Attaching rebased patch. >> > > The changes looks OK, good job there. I just found 2 issues: > > 1) 2 unit tests got broken: > > ====================================================================== > FAIL: test_service[1]: service_mod: Try to update non-existent > u'HTTP/testhost1.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM' > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest > self.test(*self.arg) > File "/root/freeipa-master/tests/test_xmlrpc/xmlrpc_test.py", line 249, in > > func = lambda: self.check(nice, **test) > File "/root/freeipa-master/tests/test_xmlrpc/xmlrpc_test.py", line 262, in check > self.check_exception(nice, cmd, args, options, expected) > File "/root/freeipa-master/tests/test_xmlrpc/xmlrpc_test.py", line 288, in > check_exception > assert_deepequal(expected.strerror, e.strerror) > File "/root/freeipa-master/tests/util.py", line 343, in assert_deepequal > VALUE % (doc, expected, got, stack) > AssertionError: assert_deepequal: expected != got. > > expected = u'HTTP/testhost1.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM: > service not found' > got = u'no such entry' > path = () > > ====================================================================== > FAIL: test_service[23]: service_mod: Try to update non-existent > u'HTTP/testhost1.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM' > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest > self.test(*self.arg) > File "/root/freeipa-master/tests/test_xmlrpc/xmlrpc_test.py", line 249, in > > func = lambda: self.check(nice, **test) > File "/root/freeipa-master/tests/test_xmlrpc/xmlrpc_test.py", line 262, in check > self.check_exception(nice, cmd, args, options, expected) > File "/root/freeipa-master/tests/test_xmlrpc/xmlrpc_test.py", line 288, in > check_exception > assert_deepequal(expected.strerror, e.strerror) > File "/root/freeipa-master/tests/util.py", line 343, in assert_deepequal > VALUE % (doc, expected, got, stack) > AssertionError: assert_deepequal: expected != got. > > expected = u'HTTP/testhost1.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM: > service not found' > got = u'no such entry' > path = () > > ---------------------------------------------------------------------- > > You would have the properly catch and process exception in service_mod plugin > in the following line: > (dn, entry_attrs_old) = ldap.get_entry(dn, ['usercertificate']) > to be able to update tests like that. > > > 2) While at internationalizing public errors, we could fix the following > exceptions in bindinstance.py. They are not supposed to be i18n-ed, nor they > are correct and will crash horribly. > > @@ -277,7 +278,7 @@ def add_zone(name, zonemgr=None, dns_backup=None, > ns_hostname=None, ns_ip_addres > # automatically retrieve list of DNS masters > dns_masters = api.Object.dnsrecord.get_dns_masters() > if not dns_masters: > - raise errors.NotFound("No IPA server with DNS support found!") > + raise errors.NotFound(_("No IPA server with DNS support found!")) > ns_main = dns_masters.pop(0) > ns_replicas = dns_masters > addresses = resolve_host(ns_main) > @@ -321,7 +322,7 @@ def add_reverse_zone(zone, ns_hostname=None, > ns_ip_address=None, > # automatically retrieve list of DNS masters > dns_masters = api.Object.dnsrecord.get_dns_masters() > if not dns_masters: > - raise errors.NotFound("No IPA server with DNS support found!") > + raise errors.NotFound(_("No IPA server with DNS support found!")) > ns_main = dns_masters.pop(0) > ns_replicas = dns_masters > addresses = resolve_host(ns_main) > > > > Martin > Thank you for the review. Made service plugin use handle_not_found, and replaced NotFound in installer code with ScriptError. -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0072-03-Internationalization-for-public-errors.patch Type: text/x-patch Size: 53239 bytes Desc: not available URL: From pviktori at redhat.com Mon Sep 3 15:56:35 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Mon, 03 Sep 2012 17:56:35 +0200 Subject: [Freeipa-devel] [PATCH 81] ipa user-find --manager does not find matches In-Reply-To: <5040C24C.7030000@redhat.com> References: <5040B58D.7030102@redhat.com> <5040C24C.7030000@redhat.com> Message-ID: <5044D333.6000501@redhat.com> On 08/31/2012 03:55 PM, John Dennis wrote: > The original patch was missing a unit test to verify the fix. This > version of the patch now includes a unit test. Works well, ACK -- Petr? From pviktori at redhat.com Mon Sep 3 16:00:21 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Mon, 03 Sep 2012 18:00:21 +0200 Subject: [Freeipa-devel] [PATCH 78] Ticket #2979 - prevent last admin from being disabled In-Reply-To: <5044C18C.5060900@redhat.com> References: <502D9291.2090102@redhat.com> <503275D9.2020907@redhat.com> <503A5A8B.1050602@redhat.com> <50449A45.3090608@redhat.com> <5044C18C.5060900@redhat.com> Message-ID: <5044D415.2030907@redhat.com> On 09/03/2012 04:41 PM, John Dennis wrote: > On 09/03/2012 07:53 AM, Petr Viktorin wrote: >> On 08/26/2012 07:19 PM, John Dennis wrote: >>> On 08/20/2012 01:37 PM, Petr Viktorin wrote: >>>> (Sorry if you're getting this twice; I didn't send it to the list) >>>> >>>> On 08/16/2012 08:38 PM, John Dennis wrote: >>>>> >>>>> -- >>>>> John Dennis >>>>> >>>>> Looking to carve out IT costs? >>>>> www.redhat.com/carveoutcosts/ >>>>> >>>>> freeipa-jdennis-0078-Ticket-2979-prevent-last-admin-from-being-disabled.patch >>>>> >>>>> >>>>> >>>>> >>>>> >From c47109c63530e188db76986fdda48c76bf681d10 Mon Sep 17 00:00:00 >>>>> 2001 >>>>> From: John Dennis >>>>> Date: Thu, 16 Aug 2012 20:28:44 -0400 >>>>> Subject: [PATCH 78] Ticket #2979 - prevent last admin from being >>>>> disabled >>>>> Content-Type: text/plain; charset="utf-8" >>>>> Content-Transfer-Encoding: 8bit >>>>> >>>>> We prevent the last member of the admin group from being deleted. The >>>>> same check needs to be performed when disabling a user. >>>>> >>>>> Moved the code in del_user to a common subroutine and call it from >>>>> both user_del and user_disable. Note, unlike user_del user_disable >>>>> does not have a 'pre' callback therefore the check function is called >>>>> in user_disable's execute routine. >>>> >>>> This should also prevent disabling all admins if there's more than one: >>>> >>>> # ipa user-add admin2 --first=a --last=b >>>> ------------------- >>>> Added user "admin2" >>>> ------------------- >>>> ... >>>> # ipa group-add-member admins --user=admin2 >>>> ------------------------- >>>> Number of members added 1 >>>> ------------------------- >>>> # ipa user-disable admin2 >>>> ------------------------------ >>>> Disabled user account "admin2" >>>> ------------------------------ >>>> # ipa user-disable admin >>>> ------------------------------ >>>> Disabled user account "admin" >>>> ------------------------------ >>>> # ipa ping >>>> ipa: ERROR: Server is unwilling to perform: Account inactivated. >>>> Contact >>>> system administrator. >>>> >>>> Also with one enabled and one disabled admin, it shouldn't be possible >>>> to delete the enabled one. >>>> >>>> >>>> Please add some tests; you can extend the ones added in commit f8e7b51. >>> >>> Good catch with respect to disabled users, thank you. >>> >>> Reworked patch attached, see patch comments. >>> >>> >>> >>> >> >> Works well now, just the error message is incorrect: it mentions only >> deleting, not disabling. >> >> $ ipa user-disable admin >> ipa: ERROR: admin cannot be deleted because it is the last member of >> group admins > > Updated the error message to say > > "... cannot be deleted or disabled because ..." > > ACK. Please push John's patch 81 before this one; that way it applies cleanly. -- Petr? From pvoborni at redhat.com Mon Sep 3 16:05:21 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Mon, 03 Sep 2012 18:05:21 +0200 Subject: [Freeipa-devel] [PATCH] 204 Update of confirmation of actions Message-ID: <5044D541.2000904@redhat.com> This patch is changing confirmation of actions according to ticket #3035, see the ticket description. It does following changes: * Confirmation of update action was removed. * Action lists resets to first action (which is usually a NOP: '-- select action --') on change of displayed entry. * New confirmation dialog was implemented. It is used for action confirmation. It is used in IPA.action to replace the call of window.confirm(message). The old call is a modal window which blocks all JS functionality and has different style than other dialogs in Web UI. The new one has same design and doesn't block background operations. https://fedorahosted.org/freeipa/ticket/3035 -- Petr Vobornik -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pvoborni-0204-Update-of-confirmation-of-actions.patch Type: text/x-patch Size: 10141 bytes Desc: not available URL: From mkosek at redhat.com Mon Sep 3 16:13:22 2012 From: mkosek at redhat.com (Martin Kosek) Date: Mon, 03 Sep 2012 18:13:22 +0200 Subject: [Freeipa-devel] [PATCH 81] ipa user-find --manager does not find matches In-Reply-To: <5044D333.6000501@redhat.com> References: <5040B58D.7030102@redhat.com> <5040C24C.7030000@redhat.com> <5044D333.6000501@redhat.com> Message-ID: <5044D722.2080108@redhat.com> On 09/03/2012 05:56 PM, Petr Viktorin wrote: > On 08/31/2012 03:55 PM, John Dennis wrote: >> The original patch was missing a unit test to verify the fix. This >> version of the patch now includes a unit test. > > Works well, ACK > Rebased and pushed to master, ipa-3-0. Martin From mkosek at redhat.com Mon Sep 3 16:14:07 2012 From: mkosek at redhat.com (Martin Kosek) Date: Mon, 03 Sep 2012 18:14:07 +0200 Subject: [Freeipa-devel] [PATCH 78] Ticket #2979 - prevent last admin from being disabled In-Reply-To: <5044D415.2030907@redhat.com> References: <502D9291.2090102@redhat.com> <503275D9.2020907@redhat.com> <503A5A8B.1050602@redhat.com> <50449A45.3090608@redhat.com> <5044C18C.5060900@redhat.com> <5044D415.2030907@redhat.com> Message-ID: <5044D74F.3060706@redhat.com> On 09/03/2012 06:00 PM, Petr Viktorin wrote: > On 09/03/2012 04:41 PM, John Dennis wrote: >> On 09/03/2012 07:53 AM, Petr Viktorin wrote: >>> On 08/26/2012 07:19 PM, John Dennis wrote: >>>> On 08/20/2012 01:37 PM, Petr Viktorin wrote: >>>>> (Sorry if you're getting this twice; I didn't send it to the list) >>>>> >>>>> On 08/16/2012 08:38 PM, John Dennis wrote: >>>>>> >>>>>> -- >>>>>> John Dennis >>>>>> >>>>>> Looking to carve out IT costs? >>>>>> www.redhat.com/carveoutcosts/ >>>>>> >>>>>> freeipa-jdennis-0078-Ticket-2979-prevent-last-admin-from-being-disabled.patch >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >From c47109c63530e188db76986fdda48c76bf681d10 Mon Sep 17 00:00:00 >>>>>> 2001 >>>>>> From: John Dennis >>>>>> Date: Thu, 16 Aug 2012 20:28:44 -0400 >>>>>> Subject: [PATCH 78] Ticket #2979 - prevent last admin from being >>>>>> disabled >>>>>> Content-Type: text/plain; charset="utf-8" >>>>>> Content-Transfer-Encoding: 8bit >>>>>> >>>>>> We prevent the last member of the admin group from being deleted. The >>>>>> same check needs to be performed when disabling a user. >>>>>> >>>>>> Moved the code in del_user to a common subroutine and call it from >>>>>> both user_del and user_disable. Note, unlike user_del user_disable >>>>>> does not have a 'pre' callback therefore the check function is called >>>>>> in user_disable's execute routine. >>>>> >>>>> This should also prevent disabling all admins if there's more than one: >>>>> >>>>> # ipa user-add admin2 --first=a --last=b >>>>> ------------------- >>>>> Added user "admin2" >>>>> ------------------- >>>>> ... >>>>> # ipa group-add-member admins --user=admin2 >>>>> ------------------------- >>>>> Number of members added 1 >>>>> ------------------------- >>>>> # ipa user-disable admin2 >>>>> ------------------------------ >>>>> Disabled user account "admin2" >>>>> ------------------------------ >>>>> # ipa user-disable admin >>>>> ------------------------------ >>>>> Disabled user account "admin" >>>>> ------------------------------ >>>>> # ipa ping >>>>> ipa: ERROR: Server is unwilling to perform: Account inactivated. >>>>> Contact >>>>> system administrator. >>>>> >>>>> Also with one enabled and one disabled admin, it shouldn't be possible >>>>> to delete the enabled one. >>>>> >>>>> >>>>> Please add some tests; you can extend the ones added in commit f8e7b51. >>>> >>>> Good catch with respect to disabled users, thank you. >>>> >>>> Reworked patch attached, see patch comments. >>>> >>>> >>>> >>>> >>> >>> Works well now, just the error message is incorrect: it mentions only >>> deleting, not disabling. >>> >>> $ ipa user-disable admin >>> ipa: ERROR: admin cannot be deleted because it is the last member of >>> group admins >> >> Updated the error message to say >> >> "... cannot be deleted or disabled because ..." >> >> > > ACK. > Please push John's patch 81 before this one; that way it applies cleanly. > Pushed to master, ipa-3-0. Martin From mkosek at redhat.com Mon Sep 3 16:17:02 2012 From: mkosek at redhat.com (Martin Kosek) Date: Mon, 03 Sep 2012 18:17:02 +0200 Subject: [Freeipa-devel] [PATCH] 0072 Internationalization for public errors In-Reply-To: <5044C944.1050509@redhat.com> References: <500FFB88.6080902@redhat.com> <50338DB9.3050606@redhat.com> <503E1D04.8070303@redhat.com> <5044C944.1050509@redhat.com> Message-ID: <5044D7FE.50506@redhat.com> On 09/03/2012 05:14 PM, Petr Viktorin wrote: > On 08/29/2012 03:45 PM, Martin Kosek wrote: >> On 08/21/2012 03:31 PM, Petr Viktorin wrote: >>> On 07/25/2012 09:58 AM, Petr Viktorin wrote: >>>> I spent some time while IPA was installing tracking down error mesages >>>> that weren't marked for translation :) >>>> I also changed the error docstrings, as people are likely to look there >>>> for inspiration. >>>> >>>> >>>> https://fedorahosted.org/freeipa/ticket/1953 >>> >>> Attaching rebased patch. >>> >> >> The changes looks OK, good job there. I just found 2 issues: >> >> 1) 2 unit tests got broken: >> >> ====================================================================== >> FAIL: test_service[1]: service_mod: Try to update non-existent >> u'HTTP/testhost1.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM' >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest >> self.test(*self.arg) >> File "/root/freeipa-master/tests/test_xmlrpc/xmlrpc_test.py", line 249, in >> >> func = lambda: self.check(nice, **test) >> File "/root/freeipa-master/tests/test_xmlrpc/xmlrpc_test.py", line 262, in >> check >> self.check_exception(nice, cmd, args, options, expected) >> File "/root/freeipa-master/tests/test_xmlrpc/xmlrpc_test.py", line 288, in >> check_exception >> assert_deepequal(expected.strerror, e.strerror) >> File "/root/freeipa-master/tests/util.py", line 343, in assert_deepequal >> VALUE % (doc, expected, got, stack) >> AssertionError: assert_deepequal: expected != got. >> >> expected = u'HTTP/testhost1.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM: >> service not found' >> got = u'no such entry' >> path = () >> >> ====================================================================== >> FAIL: test_service[23]: service_mod: Try to update non-existent >> u'HTTP/testhost1.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM' >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest >> self.test(*self.arg) >> File "/root/freeipa-master/tests/test_xmlrpc/xmlrpc_test.py", line 249, in >> >> func = lambda: self.check(nice, **test) >> File "/root/freeipa-master/tests/test_xmlrpc/xmlrpc_test.py", line 262, in >> check >> self.check_exception(nice, cmd, args, options, expected) >> File "/root/freeipa-master/tests/test_xmlrpc/xmlrpc_test.py", line 288, in >> check_exception >> assert_deepequal(expected.strerror, e.strerror) >> File "/root/freeipa-master/tests/util.py", line 343, in assert_deepequal >> VALUE % (doc, expected, got, stack) >> AssertionError: assert_deepequal: expected != got. >> >> expected = u'HTTP/testhost1.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM: >> service not found' >> got = u'no such entry' >> path = () >> >> ---------------------------------------------------------------------- >> >> You would have the properly catch and process exception in service_mod plugin >> in the following line: >> (dn, entry_attrs_old) = ldap.get_entry(dn, ['usercertificate']) >> to be able to update tests like that. >> >> >> 2) While at internationalizing public errors, we could fix the following >> exceptions in bindinstance.py. They are not supposed to be i18n-ed, nor they >> are correct and will crash horribly. >> >> @@ -277,7 +278,7 @@ def add_zone(name, zonemgr=None, dns_backup=None, >> ns_hostname=None, ns_ip_addres >> # automatically retrieve list of DNS masters >> dns_masters = api.Object.dnsrecord.get_dns_masters() >> if not dns_masters: >> - raise errors.NotFound("No IPA server with DNS support found!") >> + raise errors.NotFound(_("No IPA server with DNS support found!")) >> ns_main = dns_masters.pop(0) >> ns_replicas = dns_masters >> addresses = resolve_host(ns_main) >> @@ -321,7 +322,7 @@ def add_reverse_zone(zone, ns_hostname=None, >> ns_ip_address=None, >> # automatically retrieve list of DNS masters >> dns_masters = api.Object.dnsrecord.get_dns_masters() >> if not dns_masters: >> - raise errors.NotFound("No IPA server with DNS support found!") >> + raise errors.NotFound(_("No IPA server with DNS support found!")) >> ns_main = dns_masters.pop(0) >> ns_replicas = dns_masters >> addresses = resolve_host(ns_main) >> >> >> >> Martin >> > > Thank you for the review. > Made service plugin use handle_not_found, and replaced NotFound in installer > code with ScriptError. > ACK. Pushed to master, ipa-3-0. Martin From mkosek at redhat.com Tue Sep 4 08:27:50 2012 From: mkosek at redhat.com (Martin Kosek) Date: Tue, 04 Sep 2012 10:27:50 +0200 Subject: [Freeipa-devel] [PATCH] 299 Make replica install more robust Message-ID: <5045BB86.40707@redhat.com> This is very hard to reproduce, regression testing should be enough. I managed to reproduce the bug by installing 2 replicas in a time to a master which was busy with adding users. This patch improved a situation a bit for me in such cases. ---- Under certain circumstances, replica installation may fail in "enable GSSAPI for replication" step when it cannot sync LDAP service principals. There is often not much we can do as Directory Server may be in an unrecoverable state but we should at least wait longer before we give up. A function checking replication status was also fixed to give more accurate results by properly comparing start/end time of the replication process and returning an error message to calling function. This error message is then returned to user if do not manage to get the LDAP service principals to give him a pointer to the actual issue. https://fedorahosted.org/freeipa/ticket/2950 -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-299-make-replica-install-more-robust.patch Type: text/x-patch Size: 6174 bytes Desc: not available URL: From mkosek at redhat.com Tue Sep 4 12:06:48 2012 From: mkosek at redhat.com (Martin Kosek) Date: Tue, 04 Sep 2012 14:06:48 +0200 Subject: [Freeipa-devel] [PATCH] 298 Add safe updates for objectClasses In-Reply-To: <503F6249.3060209@redhat.com> References: <503E0DD9.2020905@redhat.com> <503F6249.3060209@redhat.com> Message-ID: <5045EED8.70005@redhat.com> On 08/30/2012 02:53 PM, Rob Crittenden wrote: > Martin Kosek wrote: >> Current objectclass updates in a form of "replace" update instruction >> dependent on exact match of the old object class specification in the >> update instruction and the real value in LDAP. However, this approach is >> very error prone as object class definition can easily differ as for >> example because of unexpected X-ORIGIN value. Such objectclass update >> failures may lead to serious malfunctions later. >> >> Add new update instruction type "replaceoc" with the following format: >> replaceoc:OID:new >> This update instruction will always replace an objectclass with >> specified OID with the new definition. >> >> https://fedorahosted.org/freeipa/ticket/2440 > > This works ok. Martin and I had a conversation in IRC about it. > > This moves from replacing a specific bit of schema with a new one, in all > cases. I wonder if we should be more conservative and know what we're replacing > in advance. > > rob > You are right, I was too harsh when replacing the objectclasses. This would cause issues when LDAP update would be run on a replica with lower version and older objectclass definitions. I came up with an alternative solution and instead of always replacing the objectclass I rather reverted to old-OC:new-OC style which should be safer. Now, the LDAP updater always normalizes an objectclass before comparing it using python-ldap objectclass model. With this approach, objectclasses differing only in X-ORIGIN or white spaces should match and be updated. Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-298-2-add-safe-updates-for-objectclasses.patch Type: text/x-patch Size: 6136 bytes Desc: not available URL: From pviktori at redhat.com Tue Sep 4 13:20:37 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Tue, 04 Sep 2012 15:20:37 +0200 Subject: [Freeipa-devel] [PATCH] 0076 Run ntpdate in verbose mode, not debug (i.e. no-op) mode Message-ID: <50460025.9010402@redhat.com> In ipa-client-install, ntpdate was passed the -d option, which makes it do nothing. Use the -v option instead. https://fedorahosted.org/freeipa/ticket/3048 -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0076-Run-ntpdate-in-verbose-mode-not-debug-i.e.-no-op-mod.patch Type: text/x-patch Size: 2722 bytes Desc: not available URL: From pviktori at redhat.com Tue Sep 4 13:22:38 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Tue, 04 Sep 2012 15:22:38 +0200 Subject: [Freeipa-devel] [PATCH] 0077 Check direct/reverse hostname/address resolution in ipa-replica-install Message-ID: <5046009E.2020306@redhat.com> https://fedorahosted.org/freeipa/ticket/2845 -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0077-Check-remote-master-hostname-address-resolution-in-i.patch Type: text/x-patch Size: 5757 bytes Desc: not available URL: From tbabej at redhat.com Tue Sep 4 13:36:46 2012 From: tbabej at redhat.com (Tomas Babej) Date: Tue, 04 Sep 2012 15:36:46 +0200 Subject: [Freeipa-devel] [PATCH 0012] Change slapi_mods_init in ipa_winsync_pre_ad_mod_user_mods_cb Message-ID: <504603EE.2000800@redhat.com> Hi, https://fedorahosted.org/freeipa/ticket/2953 Tomas. -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-tbabej-0012-Change-slapi_mods_init-in-ipa_winsync_pre_ad_mod_use.patch Type: text/x-patch Size: 1153 bytes Desc: not available URL: From mkosek at redhat.com Tue Sep 4 14:22:48 2012 From: mkosek at redhat.com (Martin Kosek) Date: Tue, 04 Sep 2012 16:22:48 +0200 Subject: [Freeipa-devel] [PATCH] 300-301 Fix DNS SOA serial parameters boundaries Message-ID: <50460EB8.2080800@redhat.com> To test, simply run the following command: ipa dnszone-mod example.com --serial=4294967295 This should work well on patched server&client. Web UI should work too as it reads the max limit dynamically. --- [PATCH 2/2] Fix DNS SOA serial parameters boundaries: Set correct boundaries for DNS SOA serial parameters (see RFC 1035, 2181). [PATCH 1/2] Transfer long numbers over XMLRPC Numeric parameters in ipalib were limited by XMLRPC boundaries for integer (2^31-1) which is too low for some LDAP attributes like DNS SOA serial field. Transfer numbers which are not in XMLRPC boundary as a string and not as a number to workaround this limitation. Int parameter had to be updated to also accept Python's long type as valid int type. -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-300-transfer-long-numbers-over-xmlrpc.patch Type: text/x-patch Size: 3592 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-301-fix-dns-soa-serial-parameters-boundaries.patch Type: text/x-patch Size: 10466 bytes Desc: not available URL: From rmeggins at redhat.com Tue Sep 4 14:40:28 2012 From: rmeggins at redhat.com (Rich Megginson) Date: Tue, 04 Sep 2012 08:40:28 -0600 Subject: [Freeipa-devel] Ticket #2866 - referential integrity in IPA In-Reply-To: <5044C1F0.7080609@redhat.com> References: <1345211089.3796.34.camel@priserak> <502E4E78.1090407@redhat.com> <503B9F4E.4010005@redhat.com> <503BA065.1060207@redhat.com> <5044C1F0.7080609@redhat.com> Message-ID: <504612DC.9060402@redhat.com> On 09/03/2012 08:42 AM, Martin Kosek wrote: > On 08/27/2012 06:29 PM, Rich Megginson wrote: >> On 08/27/2012 10:24 AM, Martin Kosek wrote: >>> On 08/17/2012 04:00 PM, Rich Megginson wrote: >>>> On 08/17/2012 07:44 AM, Martin Kosek wrote: >>>>> Hi guys, >>>>> >>>>> I am now investigating ticket #2866: >>>>> https://fedorahosted.org/freeipa/ticket/2866 >>>>> >>>>> And I am thinking about possible solutions for this problem. In a >>>>> nutshell, we do not properly check referential integrity in some IPA >>>>> objects where we keep one-way DN references to other objects, e.g. in >>>>> - managedBy attribute for a host object >>>>> - memberhost attribute for HBAC rule object >>>>> - memberuser attribute for user object >>>>> - memberallowcmd or memberdenycmd for SUDO command object (reported in >>>>> #2866) >>>>> ... >>>>> >>>>> Currently, I see 2 approaches to solve this: >>>>> 1) Add relevant checks to our ipalib plugins where problematic >>>>> operations with these operations are being executed (like we do for >>>>> selinuxusermap's seealso attribute in HBAC plugin) >>>>> This of course would not prevent direct LDAP deletes. >>>>> >>>>> 2) Implement a preop DS plugin that would hook to MODRDN and DELETE >>>>> callbacks and check that this object's DN is not referenced in other >>>>> objects. And if it does, it would reject such modification. Second >>>>> option would be to delete the attribute value with now invalid >>>>> reference. This would be probably more suitable for example for >>>>> references to user objects. >>>>> >>>>> Any comments to these possible approaches are welcome. >>>>> >>>>> Rich, do you think that as an alternative to these 2 approaches, >>>>> memberOf plugin could be eventually modified to do this task? >>>> This is very similar to the referential integrity plugin already in 389, except >>>> instead of cleaning up references to moved and deleted entries, you want it to >>>> prevent moving or deleting an entry if that entry is referenced by the >>>> managedby/memberhost/memberuser/memberallowcmd/memberdenycmd of some other >>>> entry. >>> I think that using or enhancing current DS referential integrity plugin will be >>> the best and the most consistent way to go. >>> >>> We already use that plugin for some user attributes like "manager" or >>> "secretary". seeAlso is already covered by default, so for example seeAlso >>> attribute in SELinux usermap object referencing an HBAC rule will get removed >>> when relevant HBAC rule is removed (I just checked that). >>> >>>> Note that the managed entry plugin (mep) already handles this for the managedby >>>> attribute. >>> I assume you are referencing "mepmanagedby" and "mepmanagedentry" attributes >>> which then produce errors like this one: >>> >>> # ipa netgroup-del foo >>> ipa: ERROR: Server is unwilling to perform: Deleting a managed entry is not >>> allowed. It needs to be manually unlinked first. >>> >>> managedBy attribute used by hosts objects I had in mind seems to not be covered. >>> >>> But you are right, this is pretty much what I wanted. Though in case of MEP, >>> there is a link in both referenced objects, but in our case, we have just the >>> one-way link. >>> >>>> Are you already using the memberof plugin for >>>> memberhost/memberuser/memberallowcmd/memberdenycmd? >>>> >>>> This doesn't seem like a job for memberof, this seems like more of a new check >>>> for the referential integrity plugin. >>>> >>> I am now considering if current move/delete clean up already present in >>> Referential Integrity plugin would not be sufficient for us. >>> >>> Rich, please correct me if I am wrong, but in that case, we would just need to >>> add relevant attribute names >>> (memberhost/memberuser/memberallowcmd/memberdenycmd...) to Referential >>> Integrity plugin configuration as nsslapd-pluginarg7, nsslapd-pluginarg8, ... >>> I wonder if there would be some performance issues if we add attributes to the >>> list this way. >> No, not if they are indexed for presence and equality. >> > Hello Rich, > I am back to investigate this ticket. In order to be able to deliver some > working solution to IPA 3.0, I plan to take advantage of current Referential > Integrity Plugin to clean up dangling references. > > This is the plan I plan to take: > 1) Add pres,eq indexes for all un-indexed attributes that we want to check: > sourcehost > memberservice > managedby > memberallowcmd > memberdenycmd > ipasudorunas > ipasudorunasgroup ok > > 2) Add missing "pres" index for attributes we want to check, but only have eq > index: > manager > secretary > memberuser > memberhost > > I assume this step is also needed in order to keep the server performance. yes > > 3) Add all these attributes do Referential Integrity Plugin attribute list of > not already ok > > 4) Also add Index task (nsIndexAttribute) for all these new indexes so that > they are created during IPA server upgrade. ok > > Is this procedure OK DS-wise? Yes > > I also have question regarding the following note in RHDS doc chapter 3.6. > Maintaining Referential Integrity: > > "The Referential Integrity Plug-in should only be enabled on one supplier > replica in a multi-master replication environment to avoid conflict resolution > loops..." > > Currently, we enable this plugin on all IPA replicas. Is this something we need > to be concerned about and fix ASAP (before we do all this RefInt effort)? Mark/Nathan - I know you guys have looked at this issue. > > Thank you, > Martin From rmeggins at redhat.com Tue Sep 4 15:06:14 2012 From: rmeggins at redhat.com (Rich Megginson) Date: Tue, 04 Sep 2012 09:06:14 -0600 Subject: [Freeipa-devel] [PATCH 0012] Change slapi_mods_init in ipa_winsync_pre_ad_mod_user_mods_cb In-Reply-To: <504603EE.2000800@redhat.com> References: <504603EE.2000800@redhat.com> Message-ID: <504618E6.6050400@redhat.com> On 09/04/2012 07:36 AM, Tomas Babej wrote: > Hi, > > https://fedorahosted.org/freeipa/ticket/2953 ack > > Tomas. > > > _______________________________________________ > Freeipa-devel mailing list > Freeipa-devel at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From mkosek at redhat.com Tue Sep 4 16:07:56 2012 From: mkosek at redhat.com (Martin Kosek) Date: Tue, 04 Sep 2012 18:07:56 +0200 Subject: [Freeipa-devel] [PATCH 0012] Change slapi_mods_init in ipa_winsync_pre_ad_mod_user_mods_cb In-Reply-To: <504603EE.2000800@redhat.com> References: <504603EE.2000800@redhat.com> Message-ID: <5046275C.3090604@redhat.com> On 09/04/2012 03:36 PM, Tomas Babej wrote: > Hi, > > https://fedorahosted.org/freeipa/ticket/2953 > > Tomas. > Pushed to master, ipa-3-0. Martin From mkosek at redhat.com Tue Sep 4 16:35:06 2012 From: mkosek at redhat.com (Martin Kosek) Date: Tue, 04 Sep 2012 18:35:06 +0200 Subject: [Freeipa-devel] [PATCH] 0076 Run ntpdate in verbose mode, not debug (i.e. no-op) mode In-Reply-To: <50460025.9010402@redhat.com> References: <50460025.9010402@redhat.com> Message-ID: <50462DBA.7030603@redhat.com> On 09/04/2012 03:20 PM, Petr Viktorin wrote: > In ipa-client-install, ntpdate was passed the -d option, which makes it do > nothing. Use the -v option instead. > > https://fedorahosted.org/freeipa/ticket/3048 > > ACK. Pushed to master, ipa-3-0. Martin From jcholast at redhat.com Tue Sep 4 16:54:01 2012 From: jcholast at redhat.com (Jan Cholasta) Date: Tue, 04 Sep 2012 18:54:01 +0200 Subject: [Freeipa-devel] [PATCH] 83 Use OpenSSH-style public keys as the preferred format of SSH public keys Message-ID: <50463229.5030305@redhat.com> Hi, this patch changes the format of the sshpubkey parameter to the format used by OpenSSH (see sshd(8)). Public keys in the old format (raw RFC 4253 blob) are automatically converted to OpenSSH-style public keys. OpenSSH-style public keys are now stored in LDAP. Changed sshpubkeyfp to be an output parameter, as that is what it actually is. Allow parameter normalizers to be used on values of any type, not just unicode, so that public key blobs (which are str) can be normalized to OpenSSH-style public keys. Note that you need a SSSD build including (SSSD 1.9.0beta7-to-be) in order to make OpenSSH integration actually work with OpenSSH-style public keys. Honza -- Jan Cholasta -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-jcholast-83-openssh-style-public-keys.patch Type: text/x-patch Size: 25012 bytes Desc: not available URL: From rcritten at redhat.com Tue Sep 4 17:44:06 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 04 Sep 2012 13:44:06 -0400 Subject: [Freeipa-devel] [PATCH] 0077 Check direct/reverse hostname/address resolution in ipa-replica-install In-Reply-To: <5046009E.2020306@redhat.com> References: <5046009E.2020306@redhat.com> Message-ID: <50463DE6.9070300@redhat.com> Petr Viktorin wrote: > > https://fedorahosted.org/freeipa/ticket/2845 Shouldn't this also call verify_fqdn() on the local hostname and not just the master? I think this would eventually fail in the conncheck but what if that was skipped? rob From rcritten at redhat.com Tue Sep 4 18:22:42 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 04 Sep 2012 14:22:42 -0400 Subject: [Freeipa-devel] [PATCH] 299 Make replica install more robust In-Reply-To: <5045BB86.40707@redhat.com> References: <5045BB86.40707@redhat.com> Message-ID: <504646F2.1070902@redhat.com> Martin Kosek wrote: > This is very hard to reproduce, regression testing should be enough. I managed > to reproduce the bug by installing 2 replicas in a time to a master which was > busy with adding users. > > This patch improved a situation a bit for me in such cases. > > ---- > Under certain circumstances, replica installation may fail in > "enable GSSAPI for replication" step when it cannot sync LDAP service > principals. There is often not much we can do as Directory Server > may be in an unrecoverable state but we should at least wait longer > before we give up. > > A function checking replication status was also fixed to give more > accurate results by properly comparing start/end time of the > replication process and returning an error message to calling > function. This error message is then returned to user if do not > manage to get the LDAP service principals to give him a pointer > to the actual issue. > > https://fedorahosted.org/freeipa/ticket/2950 ACK, pushed to master and ipa-3-0 rob From rcritten at redhat.com Tue Sep 4 19:04:31 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 04 Sep 2012 15:04:31 -0400 Subject: [Freeipa-devel] [PATCH 0010] Sort policies numerically in pwpolicy-find In-Reply-To: <50414AE2.6000305@redhat.com> References: <5040D9BF.6050404@redhat.com> <5040EF9A.7050904@redhat.com> <50414AE2.6000305@redhat.com> Message-ID: <504650BF.4080302@redhat.com> Tomas Babej wrote: > On 08/31/2012 07:08 PM, Rob Crittenden wrote: >> Tomas Babej wrote: >>> Hi, >>> >>> this is a fairly simple one-liner. >>> >>> https://fedorahosted.org/freeipa/ticket/3039 >>> >>> Tomas >> >> Looks good. Can you add a unit test so we don't have a regression on >> this? >> >> thanks >> >> rob >> > > I tweaked one of the existing unit tests to test the expected behaviour, > so that we are not cluttered with tests checking the same thing. > > Tomas ACK, pushed to master From rcritten at redhat.com Tue Sep 4 21:48:50 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 04 Sep 2012 17:48:50 -0400 Subject: [Freeipa-devel] [PATCH] 298 Add safe updates for objectClasses In-Reply-To: <5045EED8.70005@redhat.com> References: <503E0DD9.2020905@redhat.com> <503F6249.3060209@redhat.com> <5045EED8.70005@redhat.com> Message-ID: <50467742.5020900@redhat.com> Martin Kosek wrote: > On 08/30/2012 02:53 PM, Rob Crittenden wrote: >> Martin Kosek wrote: >>> Current objectclass updates in a form of "replace" update instruction >>> dependent on exact match of the old object class specification in the >>> update instruction and the real value in LDAP. However, this approach is >>> very error prone as object class definition can easily differ as for >>> example because of unexpected X-ORIGIN value. Such objectclass update >>> failures may lead to serious malfunctions later. >>> >>> Add new update instruction type "replaceoc" with the following format: >>> replaceoc:OID:new >>> This update instruction will always replace an objectclass with >>> specified OID with the new definition. >>> >>> https://fedorahosted.org/freeipa/ticket/2440 >> >> This works ok. Martin and I had a conversation in IRC about it. >> >> This moves from replacing a specific bit of schema with a new one, in all >> cases. I wonder if we should be more conservative and know what we're replacing >> in advance. >> >> rob >> > > You are right, I was too harsh when replacing the objectclasses. This would > cause issues when LDAP update would be run on a replica with lower version and > older objectclass definitions. > > I came up with an alternative solution and instead of always replacing the > objectclass I rather reverted to old-OC:new-OC style which should be safer. > Now, the LDAP updater always normalizes an objectclass before comparing it > using python-ldap objectclass model. With this approach, objectclasses > differing only in X-ORIGIN or white spaces should match and be updated. > > Martin > NACK I think this: + for value in replaced_values: + entry_values.remove(old) Should be: + entry_values.remove(value) I'm still doing other testing but this is what I've found so far. rob From rcritten at redhat.com Wed Sep 5 01:47:44 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 04 Sep 2012 21:47:44 -0400 Subject: [Freeipa-devel] [PATCH] 298 Add safe updates for objectClasses In-Reply-To: <50467742.5020900@redhat.com> References: <503E0DD9.2020905@redhat.com> <503F6249.3060209@redhat.com> <5045EED8.70005@redhat.com> <50467742.5020900@redhat.com> Message-ID: <5046AF40.40408@redhat.com> Rob Crittenden wrote: > Martin Kosek wrote: >> On 08/30/2012 02:53 PM, Rob Crittenden wrote: >>> Martin Kosek wrote: >>>> Current objectclass updates in a form of "replace" update instruction >>>> dependent on exact match of the old object class specification in the >>>> update instruction and the real value in LDAP. However, this >>>> approach is >>>> very error prone as object class definition can easily differ as for >>>> example because of unexpected X-ORIGIN value. Such objectclass update >>>> failures may lead to serious malfunctions later. >>>> >>>> Add new update instruction type "replaceoc" with the following format: >>>> replaceoc:OID:new >>>> This update instruction will always replace an objectclass with >>>> specified OID with the new definition. >>>> >>>> https://fedorahosted.org/freeipa/ticket/2440 >>> >>> This works ok. Martin and I had a conversation in IRC about it. >>> >>> This moves from replacing a specific bit of schema with a new one, in >>> all >>> cases. I wonder if we should be more conservative and know what we're >>> replacing >>> in advance. >>> >>> rob >>> >> >> You are right, I was too harsh when replacing the objectclasses. This >> would >> cause issues when LDAP update would be run on a replica with lower >> version and >> older objectclass definitions. >> >> I came up with an alternative solution and instead of always replacing >> the >> objectclass I rather reverted to old-OC:new-OC style which should be >> safer. >> Now, the LDAP updater always normalizes an objectclass before >> comparing it >> using python-ldap objectclass model. With this approach, objectclasses >> differing only in X-ORIGIN or white spaces should match and be updated. >> >> Martin >> > > NACK > > I think this: > > + for value in replaced_values: > + entry_values.remove(old) > > Should be: > > + entry_values.remove(value) > > I'm still doing other testing but this is what I've found so far. > > rob I did some more testing and it looks like this will do the trick. I also found a place where the schema was left as unicode and causing it to blow up inside python-ldap. Here is the diff on my working instance: diff -u ipaserver/install/ldapupdate.py /usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py --- ipaserver/install/ldapupdate.py 2012-09-04 16:59:33.210688723 -0400 +++ /usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py 2012-09-04 21:47:01.583574375 -0400 @@ -643,7 +643,7 @@ self.debug('replace: no match for replaced ObjectClass "%s"', old) continue for value in replaced_values: - entry_values.remove(old) + entry_values.remove(value) else: entry_values.remove(old) entry_values.append(new) @@ -772,7 +772,11 @@ updated = False changes = self.conn.generateModList(entry.origDataDict(), entry.toDict()) if (entry.dn == DN(('cn', 'schema'))): - updated = self.is_schema_updated(entry.toDict()) + d = dict() + e = entry.toDict() + for k,v in e.items(): + d[k] = [str(x) for x in v] + updated = self.is_schema_updated(d) else: if len(changes) >= 1: updated = True rob From rcritten at redhat.com Wed Sep 5 02:48:20 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 04 Sep 2012 22:48:20 -0400 Subject: [Freeipa-devel] [PATCH] 0075 Add nsds5ReplicaStripAttrs to replica agreements In-Reply-To: <504486B9.6070403@redhat.com> References: <504486B9.6070403@redhat.com> Message-ID: <5046BD74.3080006@redhat.com> Petr Viktorin wrote: > Generalize the fix_replica_memberof update plugin to allow updating more > replication attributes. > > Add nsds5ReplicaStripAttrs to replication agreements on update and > replica install. > > https://fedorahosted.org/freeipa/ticket/2534 ACK, pushed to master From mkosek at redhat.com Wed Sep 5 07:22:59 2012 From: mkosek at redhat.com (Martin Kosek) Date: Wed, 05 Sep 2012 09:22:59 +0200 Subject: [Freeipa-devel] [PATCH] 298 Add safe updates for objectClasses In-Reply-To: <5046AF40.40408@redhat.com> References: <503E0DD9.2020905@redhat.com> <503F6249.3060209@redhat.com> <5045EED8.70005@redhat.com> <50467742.5020900@redhat.com> <5046AF40.40408@redhat.com> Message-ID: <5046FDD3.9060200@redhat.com> On 09/05/2012 03:47 AM, Rob Crittenden wrote: > Rob Crittenden wrote: >> Martin Kosek wrote: >>> On 08/30/2012 02:53 PM, Rob Crittenden wrote: >>>> Martin Kosek wrote: >>>>> Current objectclass updates in a form of "replace" update instruction >>>>> dependent on exact match of the old object class specification in the >>>>> update instruction and the real value in LDAP. However, this >>>>> approach is >>>>> very error prone as object class definition can easily differ as for >>>>> example because of unexpected X-ORIGIN value. Such objectclass update >>>>> failures may lead to serious malfunctions later. >>>>> >>>>> Add new update instruction type "replaceoc" with the following format: >>>>> replaceoc:OID:new >>>>> This update instruction will always replace an objectclass with >>>>> specified OID with the new definition. >>>>> >>>>> https://fedorahosted.org/freeipa/ticket/2440 >>>> >>>> This works ok. Martin and I had a conversation in IRC about it. >>>> >>>> This moves from replacing a specific bit of schema with a new one, in >>>> all >>>> cases. I wonder if we should be more conservative and know what we're >>>> replacing >>>> in advance. >>>> >>>> rob >>>> >>> >>> You are right, I was too harsh when replacing the objectclasses. This >>> would >>> cause issues when LDAP update would be run on a replica with lower >>> version and >>> older objectclass definitions. >>> >>> I came up with an alternative solution and instead of always replacing >>> the >>> objectclass I rather reverted to old-OC:new-OC style which should be >>> safer. >>> Now, the LDAP updater always normalizes an objectclass before >>> comparing it >>> using python-ldap objectclass model. With this approach, objectclasses >>> differing only in X-ORIGIN or white spaces should match and be updated. >>> >>> Martin >>> >> >> NACK >> >> I think this: >> >> + for value in replaced_values: >> + entry_values.remove(old) >> >> Should be: >> >> + entry_values.remove(value) Right. Thanks for the fix. It only worked in my testing because I had no objectclass update which would differ in X-ORIGIN or whitespaces or case. I tried to mangle an update file and with this fix it did the update even if X-ORIGIN and whitespaces differed. >> >> I'm still doing other testing but this is what I've found so far. >> >> rob > > I did some more testing and it looks like this will do the trick. > > I also found a place where the schema was left as unicode and causing it to > blow up inside python-ldap. Here is the diff on my working instance: > > diff -u ipaserver/install/ldapupdate.py > /usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py > --- ipaserver/install/ldapupdate.py 2012-09-04 16:59:33.210688723 -0400 > +++ /usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py 2012-09-04 > 21:47:01.583574375 -0400 > @@ -643,7 +643,7 @@ > self.debug('replace: no match for replaced > ObjectClass "%s"', old) > continue > for value in replaced_values: > - entry_values.remove(old) > + entry_values.remove(value) > else: > entry_values.remove(old) > entry_values.append(new) > @@ -772,7 +772,11 @@ > updated = False > changes = self.conn.generateModList(entry.origDataDict(), > entry.toDict()) > if (entry.dn == DN(('cn', 'schema'))): > - updated = self.is_schema_updated(entry.toDict()) > + d = dict() > + e = entry.toDict() > + for k,v in e.items(): > + d[k] = [str(x) for x in v] > + updated = self.is_schema_updated(d) > else: > if len(changes) >= 1: > updated = True > > rob > I did not hit this error during testing, but at least it won't harm. Sending an updated patch. -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-298-3-add-safe-updates-for-objectclasses.patch Type: text/x-patch Size: 6758 bytes Desc: not available URL: From pviktori at redhat.com Wed Sep 5 07:28:14 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Wed, 05 Sep 2012 09:28:14 +0200 Subject: [Freeipa-devel] [PATCH] 0077 Check direct/reverse hostname/address resolution in ipa-replica-install In-Reply-To: <50463DE6.9070300@redhat.com> References: <5046009E.2020306@redhat.com> <50463DE6.9070300@redhat.com> Message-ID: <5046FF0E.1000504@redhat.com> On 09/04/2012 07:44 PM, Rob Crittenden wrote: > Petr Viktorin wrote: >> >> https://fedorahosted.org/freeipa/ticket/2845 > > Shouldn't this also call verify_fqdn() on the local hostname and not > just the master? I think this would eventually fail in the conncheck but > what if that was skipped? > > rob A few lines above there is a call to get_host_name, which will call verify_fqdn. -- Petr? From mkosek at redhat.com Wed Sep 5 08:04:14 2012 From: mkosek at redhat.com (Martin Kosek) Date: Wed, 05 Sep 2012 10:04:14 +0200 Subject: [Freeipa-devel] [PATCH] 302 Stricter IP network validator in dnszone-add command Message-ID: <5047077E.1040000@redhat.com> We allowed IP addresses without network specification which lead to unexpected results when the zone was being created. We should rather strictly require the prefix/netmask specifying the IP network that the reverse zone should be created for. This is already done in Web UI. A unit test exercising this new validation was added. https://fedorahosted.org/freeipa/ticket/2461 -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-302-stricter-ip-network-validator-in-dnszone-add-command.patch Type: text/x-patch Size: 2726 bytes Desc: not available URL: From pvoborni at redhat.com Wed Sep 5 08:14:07 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Wed, 05 Sep 2012 10:14:07 +0200 Subject: [Freeipa-devel] [PATCH] 205 Reflect API change of SSH store in Web UI Message-ID: <504709CF.4010201@redhat.com> Format of ipasshpubkey in users and hosts changed from BYTES to STR. Web UI no longer gets the value as base64 encoded string in an object. Label was changed to reflect that the key don't have to be plain base64 encoded blob. https://fedorahosted.org/freeipa/ticket/2989 Note: freeipa-jcholast-83-openssh-style-public-keys.patch should be applied -- Petr Vobornik -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pvoborni-0205-Reflect-API-change-of-SSH-store-in-Web-UI.patch Type: text/x-patch Size: 9131 bytes Desc: not available URL: From jcholast at redhat.com Wed Sep 5 09:30:37 2012 From: jcholast at redhat.com (Jan Cholasta) Date: Wed, 05 Sep 2012 11:30:37 +0200 Subject: [Freeipa-devel] [PATCH] 302 Stricter IP network validator in dnszone-add command In-Reply-To: <5047077E.1040000@redhat.com> References: <5047077E.1040000@redhat.com> Message-ID: <50471BBD.1070506@redhat.com> Dne 5.9.2012 10:04, Martin Kosek napsal(a): > We allowed IP addresses without network specification which lead > to unexpected results when the zone was being created. We should rather > strictly require the prefix/netmask specifying the IP network that > the reverse zone should be created for. This is already done in > Web UI. > > A unit test exercising this new validation was added. > > https://fedorahosted.org/freeipa/ticket/2461 > I don't like this much. I would suggest using CheckedIPAddress and not forcing the user to enter the prefix length instead. CheckedIPAddress uses a sensible default prefix length if one is not specified (class-based for IPv4, /64 for IPv6) as opposed to IPNetwork (/32 for IPv4, /128 for IPv6 - this causes the erroneous reverse zones to be created as described in the ticket). Honza -- Jan Cholasta From pviktori at redhat.com Wed Sep 5 10:14:17 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Wed, 05 Sep 2012 12:14:17 +0200 Subject: [Freeipa-devel] [PATCH] 300-301 Fix DNS SOA serial parameters boundaries In-Reply-To: <50460EB8.2080800@redhat.com> References: <50460EB8.2080800@redhat.com> Message-ID: <504725F9.3030509@redhat.com> This works well, but please see some comments below. On 09/04/2012 04:22 PM, Martin Kosek wrote: > To test, simply run the following command: > > ipa dnszone-mod example.com --serial=4294967295 > > This should work well on patched server&client. Web UI should work too as it > reads the max limit dynamically. Please put this in the test suite. > --- > [PATCH 2/2] Fix DNS SOA serial parameters boundaries: > > Set correct boundaries for DNS SOA serial parameters (see RFC 1035, > 2181). > > > [PATCH 1/2] Transfer long numbers over XMLRPC > > Numeric parameters in ipalib were limited by XMLRPC boundaries for > integer (2^31-1) which is too low for some LDAP attributes like DNS > SOA serial field. > > Transfer numbers which are not in XMLRPC boundary as a string and not > as a number to workaround this limitation. Int parameter had to be > updated to also accept Python's long type as valid int type. > > > freeipa-mkosek-300-transfer-long-numbers-over-xmlrpc.patch > > >>From 8782015a17b130c5ebae8b014a7241810b10dedd Mon Sep 17 00:00:00 2001 > From: Martin Kosek > Date: Tue, 4 Sep 2012 15:49:26 +0200 > Subject: [PATCH 1/2] Transfer long numbers over XMLRPC > > Numeric parameters in ipalib were limited by XMLRPC boundaries for > integer (2^31-1) which is too low for some LDAP attributes like DNS > SOA serial field. > > Transfer numbers which are not in XMLRPC boundary as a string and not > as a number to workaround this limitation. Int parameter had to be > updated to also accept Python's long type as valid int type. > > https://fedorahosted.org/freeipa/ticket/2568 > --- > ipalib/parameters.py | 12 ++++++------ > ipalib/rpc.py | 5 ++++- > 2 files changed, 10 insertions(+), 7 deletions(-) > > diff --git a/ipalib/parameters.py b/ipalib/parameters.py > index de0d14faf08d1ab79c99e65dab9cc08f406e3a1d..21e30356b2a351bf7a3be7d47d7fabf0130cf6d4 100644 > --- a/ipalib/parameters.py > +++ b/ipalib/parameters.py > @@ -1077,7 +1077,7 @@ class Number(Param): > """ > if type(value) is self.type: > return value > - if type(value) in (unicode, int, float): > + if type(value) in (unicode, int, long, float): PEP8 says that "Object type comparisons should always use isinstance() instead of comparing types directly". It would be nice to change the old code whenever it is touched. It's also in a few more places in the patch. > diff --git a/ipalib/rpc.py b/ipalib/rpc.py > index d1764e3e30492d5855450398e86689bfcad7aa39..85239ac65903acf447a4d971cce70f819979ce8d 100644 > --- a/ipalib/rpc.py > +++ b/ipalib/rpc.py > @@ -94,6 +95,8 @@ def xml_wrap(value): > if type(value) is Decimal: > # transfer Decimal as a string > return unicode(value) > + if isinstance(value, (int, long)) and (value < MININT or value > MAXINT): > + return unicode(value) > if isinstance(value, DN): > return str(value) > assert type(value) in (unicode, int, float, bool, NoneType) `long` should also be included here. >>> api.Command.user_find(uidnumber=1511000000) {'count': 1, 'truncated': False, 'result': ({...},), 'summary': u'1 user matched'} >>> api.Command.user_find(uidnumber=1511000000 + 0L) Traceback (most recent call last): File "", line 1, in ... File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 102, in xml_wrap assert type(value) in (unicode, int, float, bool, NoneType) AssertionError -- Petr? From pspacek at redhat.com Wed Sep 5 10:22:29 2012 From: pspacek at redhat.com (Petr Spacek) Date: Wed, 05 Sep 2012 12:22:29 +0200 Subject: [Freeipa-devel] [PATCH] 302 Stricter IP network validator in dnszone-add command In-Reply-To: <50471BBD.1070506@redhat.com> References: <5047077E.1040000@redhat.com> <50471BBD.1070506@redhat.com> Message-ID: <504727E5.5040101@redhat.com> On 09/05/2012 11:30 AM, Jan Cholasta wrote: > Dne 5.9.2012 10:04, Martin Kosek napsal(a): >> We allowed IP addresses without network specification which lead >> to unexpected results when the zone was being created. We should rather >> strictly require the prefix/netmask specifying the IP network that >> the reverse zone should be created for. This is already done in >> Web UI. >> >> A unit test exercising this new validation was added. >> >> https://fedorahosted.org/freeipa/ticket/2461 >> > > I don't like this much. I would suggest using CheckedIPAddress and not forcing > the user to enter the prefix length instead. > > CheckedIPAddress uses a sensible default prefix length if one is not specified > (class-based for IPv4, /64 for IPv6) as opposed to IPNetwork (/32 for IPv4, > /128 for IPv6 - this causes the erroneous reverse zones to be created as > described in the ticket). > Hello, I don't like automatic netmask guessing. I have met class-based guessing in Windows (XP?) and I was forced to overwrite default mask all the time ... IMHO there is no "sensible default prefix" in real world. I sitting on network with /23 prefix right now. Also, I have never seen 10.x network with /8 prefix. -- Petr^2 Spacek From pviktori at redhat.com Wed Sep 5 10:26:33 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Wed, 05 Sep 2012 12:26:33 +0200 Subject: [Freeipa-devel] [PATCH] 300-301 Fix DNS SOA serial parameters boundaries In-Reply-To: <504725F9.3030509@redhat.com> References: <50460EB8.2080800@redhat.com> <504725F9.3030509@redhat.com> Message-ID: <504728D9.7020807@redhat.com> On 09/05/2012 12:14 PM, Petr Viktorin wrote: > This works well, but please see some comments below. > > On 09/04/2012 04:22 PM, Martin Kosek wrote: >> To test, simply run the following command: >> >> ipa dnszone-mod example.com --serial=4294967295 >> >> This should work well on patched server&client. Web UI should work too >> as it >> reads the max limit dynamically. > > Please put this in the test suite. > >> --- >> [PATCH 2/2] Fix DNS SOA serial parameters boundaries: >> >> Set correct boundaries for DNS SOA serial parameters (see RFC 1035, >> 2181). >> >> >> [PATCH 1/2] Transfer long numbers over XMLRPC >> >> Numeric parameters in ipalib were limited by XMLRPC boundaries for >> integer (2^31-1) which is too low for some LDAP attributes like DNS >> SOA serial field. >> >> Transfer numbers which are not in XMLRPC boundary as a string and not >> as a number to workaround this limitation. Int parameter had to be >> updated to also accept Python's long type as valid int type. >> >> >> freeipa-mkosek-300-transfer-long-numbers-over-xmlrpc.patch >> >> >>> From 8782015a17b130c5ebae8b014a7241810b10dedd Mon Sep 17 00:00:00 2001 >> From: Martin Kosek >> Date: Tue, 4 Sep 2012 15:49:26 +0200 >> Subject: [PATCH 1/2] Transfer long numbers over XMLRPC >> >> Numeric parameters in ipalib were limited by XMLRPC boundaries for >> integer (2^31-1) which is too low for some LDAP attributes like DNS >> SOA serial field. >> >> Transfer numbers which are not in XMLRPC boundary as a string and not >> as a number to workaround this limitation. Int parameter had to be >> updated to also accept Python's long type as valid int type. >> >> https://fedorahosted.org/freeipa/ticket/2568 >> --- >> ipalib/parameters.py | 12 ++++++------ >> ipalib/rpc.py | 5 ++++- >> 2 files changed, 10 insertions(+), 7 deletions(-) >> >> diff --git a/ipalib/parameters.py b/ipalib/parameters.py >> index >> de0d14faf08d1ab79c99e65dab9cc08f406e3a1d..21e30356b2a351bf7a3be7d47d7fabf0130cf6d4 >> 100644 >> --- a/ipalib/parameters.py >> +++ b/ipalib/parameters.py >> @@ -1077,7 +1077,7 @@ class Number(Param): >> """ >> if type(value) is self.type: >> return value >> - if type(value) in (unicode, int, float): >> + if type(value) in (unicode, int, long, float): > > > PEP8 says that "Object type comparisons should always use isinstance() > instead of comparing types directly". > It would be nice to change the old code whenever it is touched. It's > also in a few more places in the patch. > >> diff --git a/ipalib/rpc.py b/ipalib/rpc.py >> index >> d1764e3e30492d5855450398e86689bfcad7aa39..85239ac65903acf447a4d971cce70f819979ce8d >> 100644 >> --- a/ipalib/rpc.py >> +++ b/ipalib/rpc.py >> @@ -94,6 +95,8 @@ def xml_wrap(value): >> if type(value) is Decimal: >> # transfer Decimal as a string >> return unicode(value) >> + if isinstance(value, (int, long)) and (value < MININT or value > >> MAXINT): >> + return unicode(value) >> if isinstance(value, DN): >> return str(value) >> assert type(value) in (unicode, int, float, bool, NoneType) > > `long` should also be included here. > > >>> api.Command.user_find(uidnumber=1511000000) > {'count': 1, 'truncated': False, 'result': ({...},), 'summary': u'1 user > matched'} > >>> api.Command.user_find(uidnumber=1511000000 + 0L) > Traceback (most recent call last): > File "", line 1, in > ... > File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 102, in > xml_wrap > assert type(value) in (unicode, int, float, bool, NoneType) > AssertionError > > > > One more thing: please update the VERSION file. -- Petr? From mkosek at redhat.com Wed Sep 5 10:26:43 2012 From: mkosek at redhat.com (Martin Kosek) Date: Wed, 05 Sep 2012 12:26:43 +0200 Subject: [Freeipa-devel] [PATCH] 302 Stricter IP network validator in dnszone-add command In-Reply-To: <504727E5.5040101@redhat.com> References: <5047077E.1040000@redhat.com> <50471BBD.1070506@redhat.com> <504727E5.5040101@redhat.com> Message-ID: <504728E3.9060307@redhat.com> On 09/05/2012 12:22 PM, Petr Spacek wrote: > On 09/05/2012 11:30 AM, Jan Cholasta wrote: >> Dne 5.9.2012 10:04, Martin Kosek napsal(a): >>> We allowed IP addresses without network specification which lead >>> to unexpected results when the zone was being created. We should rather >>> strictly require the prefix/netmask specifying the IP network that >>> the reverse zone should be created for. This is already done in >>> Web UI. >>> >>> A unit test exercising this new validation was added. >>> >>> https://fedorahosted.org/freeipa/ticket/2461 >>> >> >> I don't like this much. I would suggest using CheckedIPAddress and not forcing >> the user to enter the prefix length instead. >> >> CheckedIPAddress uses a sensible default prefix length if one is not specified >> (class-based for IPv4, /64 for IPv6) as opposed to IPNetwork (/32 for IPv4, >> /128 for IPv6 - this causes the erroneous reverse zones to be created as >> described in the ticket). >> > Hello, > > I don't like automatic netmask guessing. I have met class-based guessing in > Windows (XP?) and I was forced to overwrite default mask all the time ... > > IMHO there is no "sensible default prefix" in real world. I sitting on network > with /23 prefix right now. Also, I have never seen 10.x network with /8 prefix. > +1 I would rather force user to choose the netmask and receive an expectable result that to be confused with CIDR-based (for IPv6) values. Martin From jcholast at redhat.com Wed Sep 5 10:36:12 2012 From: jcholast at redhat.com (Jan Cholasta) Date: Wed, 05 Sep 2012 12:36:12 +0200 Subject: [Freeipa-devel] [PATCH] 302 Stricter IP network validator in dnszone-add command In-Reply-To: <504727E5.5040101@redhat.com> References: <5047077E.1040000@redhat.com> <50471BBD.1070506@redhat.com> <504727E5.5040101@redhat.com> Message-ID: <50472B1C.2070207@redhat.com> Dne 5.9.2012 12:22, Petr Spacek napsal(a): > On 09/05/2012 11:30 AM, Jan Cholasta wrote: >> Dne 5.9.2012 10:04, Martin Kosek napsal(a): >>> We allowed IP addresses without network specification which lead >>> to unexpected results when the zone was being created. We should rather >>> strictly require the prefix/netmask specifying the IP network that >>> the reverse zone should be created for. This is already done in >>> Web UI. >>> >>> A unit test exercising this new validation was added. >>> >>> https://fedorahosted.org/freeipa/ticket/2461 >>> >> >> I don't like this much. I would suggest using CheckedIPAddress and not >> forcing >> the user to enter the prefix length instead. >> >> CheckedIPAddress uses a sensible default prefix length if one is not >> specified >> (class-based for IPv4, /64 for IPv6) as opposed to IPNetwork (/32 for >> IPv4, >> /128 for IPv6 - this causes the erroneous reverse zones to be created as >> described in the ticket). >> > Hello, > > I don't like automatic netmask guessing. I have met class-based guessing > in Windows (XP?) and I was forced to overwrite default mask all the time > ... If there was no guessing, you would have to write the netmask anyway, so I don't see any harm in guessing here. > > IMHO there is no "sensible default prefix" in real world. I sitting on > network with /23 prefix right now. Also, I have never seen 10.x network > with /8 prefix. > While this might be true for IPv4 in some cases, /64 is perfectly sensible for IPv6. Also, I have never seen 192.168.x.x network with non-/24 prefix. Honza -- Jan Cholasta From mkosek at redhat.com Wed Sep 5 10:48:22 2012 From: mkosek at redhat.com (Martin Kosek) Date: Wed, 05 Sep 2012 12:48:22 +0200 Subject: [Freeipa-devel] [PATCH] 302 Stricter IP network validator in dnszone-add command In-Reply-To: <50472B1C.2070207@redhat.com> References: <5047077E.1040000@redhat.com> <50471BBD.1070506@redhat.com> <504727E5.5040101@redhat.com> <50472B1C.2070207@redhat.com> Message-ID: <50472DF6.8000207@redhat.com> On 09/05/2012 12:36 PM, Jan Cholasta wrote: > Dne 5.9.2012 12:22, Petr Spacek napsal(a): >> On 09/05/2012 11:30 AM, Jan Cholasta wrote: >>> Dne 5.9.2012 10:04, Martin Kosek napsal(a): >>>> We allowed IP addresses without network specification which lead >>>> to unexpected results when the zone was being created. We should rather >>>> strictly require the prefix/netmask specifying the IP network that >>>> the reverse zone should be created for. This is already done in >>>> Web UI. >>>> >>>> A unit test exercising this new validation was added. >>>> >>>> https://fedorahosted.org/freeipa/ticket/2461 >>>> >>> >>> I don't like this much. I would suggest using CheckedIPAddress and not >>> forcing >>> the user to enter the prefix length instead. >>> >>> CheckedIPAddress uses a sensible default prefix length if one is not >>> specified >>> (class-based for IPv4, /64 for IPv6) as opposed to IPNetwork (/32 for >>> IPv4, >>> /128 for IPv6 - this causes the erroneous reverse zones to be created as >>> described in the ticket). >>> >> Hello, >> >> I don't like automatic netmask guessing. I have met class-based guessing >> in Windows (XP?) and I was forced to overwrite default mask all the time >> ... > > If there was no guessing, you would have to write the netmask anyway, so I > don't see any harm in guessing here. > >> >> IMHO there is no "sensible default prefix" in real world. I sitting on >> network with /23 prefix right now. Also, I have never seen 10.x network >> with /8 prefix. >> > > While this might be true for IPv4 in some cases, /64 is perfectly sensible for > IPv6. Also, I have never seen 192.168.x.x network with non-/24 prefix. > > Honza > While this may be true for 192.168.x.x, it does not apply for 10.x.x.x networks as Petr already pointed out. I don't think that there will be many people expecting that a reverse zone of 10.0.0.0/24 would be created. And since FreeIPA is mainly deployed to internal networks, I assume this will be the case of most users. Martin From atkac at redhat.com Wed Sep 5 10:58:41 2012 From: atkac at redhat.com (Adam Tkac) Date: Wed, 5 Sep 2012 12:58:41 +0200 Subject: [Freeipa-devel] [PATCH] 302 Stricter IP network validator in dnszone-add command In-Reply-To: <50472DF6.8000207@redhat.com> References: <5047077E.1040000@redhat.com> <50471BBD.1070506@redhat.com> <504727E5.5040101@redhat.com> <50472B1C.2070207@redhat.com> <50472DF6.8000207@redhat.com> Message-ID: <20120905105840.GA2549@redhat.com> On Wed, Sep 05, 2012 at 12:48:22PM +0200, Martin Kosek wrote: > On 09/05/2012 12:36 PM, Jan Cholasta wrote: > > Dne 5.9.2012 12:22, Petr Spacek napsal(a): > >> On 09/05/2012 11:30 AM, Jan Cholasta wrote: > >>> Dne 5.9.2012 10:04, Martin Kosek napsal(a): > >>>> We allowed IP addresses without network specification which lead > >>>> to unexpected results when the zone was being created. We should rather > >>>> strictly require the prefix/netmask specifying the IP network that > >>>> the reverse zone should be created for. This is already done in > >>>> Web UI. > >>>> > >>>> A unit test exercising this new validation was added. > >>>> > >>>> https://fedorahosted.org/freeipa/ticket/2461 > >>>> > >>> > >>> I don't like this much. I would suggest using CheckedIPAddress and not > >>> forcing > >>> the user to enter the prefix length instead. > >>> > >>> CheckedIPAddress uses a sensible default prefix length if one is not > >>> specified > >>> (class-based for IPv4, /64 for IPv6) as opposed to IPNetwork (/32 for > >>> IPv4, > >>> /128 for IPv6 - this causes the erroneous reverse zones to be created as > >>> described in the ticket). > >>> > >> Hello, > >> > >> I don't like automatic netmask guessing. I have met class-based guessing > >> in Windows (XP?) and I was forced to overwrite default mask all the time > >> ... > > > > If there was no guessing, you would have to write the netmask anyway, so I > > don't see any harm in guessing here. > > > >> > >> IMHO there is no "sensible default prefix" in real world. I sitting on > >> network with /23 prefix right now. Also, I have never seen 10.x network > >> with /8 prefix. > >> > > > > While this might be true for IPv4 in some cases, /64 is perfectly sensible for > > IPv6. Also, I have never seen 192.168.x.x network with non-/24 prefix. > > > > Honza > > > > While this may be true for 192.168.x.x, it does not apply for 10.x.x.x networks > as Petr already pointed out. I don't think that there will be many people > expecting that a reverse zone of 10.0.0.0/24 would be created. +1 for explicit netmasks. Although 192.168.X.0/24 is common for home networks, it's not common for company networks. When company use 192.168.0.0/16 network, it is nearly always splitted into something with for example 255.255.240.0 netmask because 255 machines in one network is too low number. Regards, Adam -- Adam Tkac, Red Hat, Inc. From jcholast at redhat.com Wed Sep 5 11:02:35 2012 From: jcholast at redhat.com (Jan Cholasta) Date: Wed, 05 Sep 2012 13:02:35 +0200 Subject: [Freeipa-devel] [PATCH] 302 Stricter IP network validator in dnszone-add command In-Reply-To: <50472DF6.8000207@redhat.com> References: <5047077E.1040000@redhat.com> <50471BBD.1070506@redhat.com> <504727E5.5040101@redhat.com> <50472B1C.2070207@redhat.com> <50472DF6.8000207@redhat.com> Message-ID: <5047314B.30201@redhat.com> Dne 5.9.2012 12:48, Martin Kosek napsal(a): > On 09/05/2012 12:36 PM, Jan Cholasta wrote: >> Dne 5.9.2012 12:22, Petr Spacek napsal(a): >>> On 09/05/2012 11:30 AM, Jan Cholasta wrote: >>>> Dne 5.9.2012 10:04, Martin Kosek napsal(a): >>>>> We allowed IP addresses without network specification which lead >>>>> to unexpected results when the zone was being created. We should rather >>>>> strictly require the prefix/netmask specifying the IP network that >>>>> the reverse zone should be created for. This is already done in >>>>> Web UI. >>>>> >>>>> A unit test exercising this new validation was added. >>>>> >>>>> https://fedorahosted.org/freeipa/ticket/2461 >>>>> >>>> >>>> I don't like this much. I would suggest using CheckedIPAddress and not >>>> forcing >>>> the user to enter the prefix length instead. >>>> >>>> CheckedIPAddress uses a sensible default prefix length if one is not >>>> specified >>>> (class-based for IPv4, /64 for IPv6) as opposed to IPNetwork (/32 for >>>> IPv4, >>>> /128 for IPv6 - this causes the erroneous reverse zones to be created as >>>> described in the ticket). >>>> >>> Hello, >>> >>> I don't like automatic netmask guessing. I have met class-based guessing >>> in Windows (XP?) and I was forced to overwrite default mask all the time >>> ... >> >> If there was no guessing, you would have to write the netmask anyway, so I >> don't see any harm in guessing here. >> >>> >>> IMHO there is no "sensible default prefix" in real world. I sitting on >>> network with /23 prefix right now. Also, I have never seen 10.x network >>> with /8 prefix. >>> >> >> While this might be true for IPv4 in some cases, /64 is perfectly sensible for >> IPv6. Also, I have never seen 192.168.x.x network with non-/24 prefix. >> >> Honza >> > > While this may be true for 192.168.x.x, it does not apply for 10.x.x.x networks > as Petr already pointed out. I don't think that there will be many people > expecting that a reverse zone of 10.0.0.0/24 would be created. And they would be correct, because the default prefix length for a class A network is /8, not /24. > > And since FreeIPA is mainly deployed to internal networks, I assume this will > be the case of most users. > > Martin > OK, but what about IPv6? Correct me if I'm wrong, but the prefix length is going to be /64 99% of the time for IPv6. The installer uses /24 for IPv4 addresses and /64 for IPv6 addresses, maybe this should be used as a default here as well. Honza -- Jan Cholasta From atkac at redhat.com Wed Sep 5 11:02:23 2012 From: atkac at redhat.com (Adam Tkac) Date: Wed, 5 Sep 2012 13:02:23 +0200 Subject: [Freeipa-devel] [PATCH 0050] Fix memory leak in configuration with multiple LDAP instances In-Reply-To: <502A59F5.90300@redhat.com> References: <502A59F5.90300@redhat.com> Message-ID: <20120905110222.GB2549@redhat.com> On Tue, Aug 14, 2012 at 04:00:21PM +0200, Petr Spacek wrote: > Hello, > > this patch fixes $SUBJ$. > > Adam, please double-check correctness of this change. > > I had two assumptions: > - all locking is done inside dns_db_(un)register() functions > - LDAP instances are decommissioned before dynamic_driver_destroy() call Ack > From e314eb7da7bfbbb2ae9d4ce1252d886c9a744e7f Mon Sep 17 00:00:00 2001 > From: Petr Spacek > Date: Tue, 14 Aug 2012 15:53:42 +0200 > Subject: [PATCH] Fix memory leak in configuration with multiple LDAP > instances. > > Signed-off-by: Petr Spacek > --- > src/ldap_driver.c | 6 ++++-- > 1 file changed, 4 insertions(+), 2 deletions(-) > > diff --git a/src/ldap_driver.c b/src/ldap_driver.c > index d958d15bdebe5e89dc4948655ffba655073d53e0..117215ae480cdcabb2977037af9a89bb25578243 100644 > --- a/src/ldap_driver.c > +++ b/src/ldap_driver.c > @@ -1278,6 +1278,7 @@ isc_result_t > dynamic_driver_init(isc_mem_t *mctx, const char *name, const char * const *argv, > dns_dyndb_arguments_t *dyndb_args) > { > + dns_dbimplementation_t *ldapdb_imp_new = NULL; > isc_result_t result; > > REQUIRE(name != NULL); > @@ -1305,11 +1306,12 @@ dynamic_driver_init(isc_mem_t *mctx, const char *name, const char * const *argv, > } > > /* Register new DNS DB implementation. */ > - ldapdb_imp = NULL; > result = dns_db_register(ldapdb_impname, &ldapdb_create, NULL, mctx, > - &ldapdb_imp); > + &ldapdb_imp_new); > if (result != ISC_R_SUCCESS && result != ISC_R_EXISTS) > return result; > + else if (result == ISC_R_SUCCESS) > + ldapdb_imp = ldapdb_imp_new; > > /* Finally, create the instance. */ > result = manager_create_db_instance(mctx, name, argv, dyndb_args); > -- > 1.7.11.2 > -- Adam Tkac, Red Hat, Inc. From atkac at redhat.com Wed Sep 5 11:06:36 2012 From: atkac at redhat.com (Adam Tkac) Date: Wed, 5 Sep 2012 13:06:36 +0200 Subject: [Freeipa-devel] [PATCH] 302 Stricter IP network validator in dnszone-add command In-Reply-To: <5047314B.30201@redhat.com> References: <5047077E.1040000@redhat.com> <50471BBD.1070506@redhat.com> <504727E5.5040101@redhat.com> <50472B1C.2070207@redhat.com> <50472DF6.8000207@redhat.com> <5047314B.30201@redhat.com> Message-ID: <20120905110635.GC2549@redhat.com> On Wed, Sep 05, 2012 at 01:02:35PM +0200, Jan Cholasta wrote: > Dne 5.9.2012 12:48, Martin Kosek napsal(a): > >On 09/05/2012 12:36 PM, Jan Cholasta wrote: > >>Dne 5.9.2012 12:22, Petr Spacek napsal(a): > >>>On 09/05/2012 11:30 AM, Jan Cholasta wrote: > >>>>Dne 5.9.2012 10:04, Martin Kosek napsal(a): > >>>>>We allowed IP addresses without network specification which lead > >>>>>to unexpected results when the zone was being created. We should rather > >>>>>strictly require the prefix/netmask specifying the IP network that > >>>>>the reverse zone should be created for. This is already done in > >>>>>Web UI. > >>>>> > >>>>>A unit test exercising this new validation was added. > >>>>> > >>>>>https://fedorahosted.org/freeipa/ticket/2461 > >>>>> > >>>> > >>>>I don't like this much. I would suggest using CheckedIPAddress and not > >>>>forcing > >>>>the user to enter the prefix length instead. > >>>> > >>>>CheckedIPAddress uses a sensible default prefix length if one is not > >>>>specified > >>>>(class-based for IPv4, /64 for IPv6) as opposed to IPNetwork (/32 for > >>>>IPv4, > >>>>/128 for IPv6 - this causes the erroneous reverse zones to be created as > >>>>described in the ticket). > >>>> > >>>Hello, > >>> > >>>I don't like automatic netmask guessing. I have met class-based guessing > >>>in Windows (XP?) and I was forced to overwrite default mask all the time > >>>... > >> > >>If there was no guessing, you would have to write the netmask anyway, so I > >>don't see any harm in guessing here. > >> > >>> > >>>IMHO there is no "sensible default prefix" in real world. I sitting on > >>>network with /23 prefix right now. Also, I have never seen 10.x network > >>>with /8 prefix. > >>> > >> > >>While this might be true for IPv4 in some cases, /64 is perfectly sensible for > >>IPv6. Also, I have never seen 192.168.x.x network with non-/24 prefix. > >> > >>Honza > >> > > > >While this may be true for 192.168.x.x, it does not apply for 10.x.x.x networks > >as Petr already pointed out. I don't think that there will be many people > >expecting that a reverse zone of 10.0.0.0/24 would be created. > > And they would be correct, because the default prefix length for a > class A network is /8, not /24. > > > > >And since FreeIPA is mainly deployed to internal networks, I assume this will > >be the case of most users. > > > >Martin > > > > OK, but what about IPv6? Correct me if I'm wrong, but the prefix > length is going to be /64 99% of the time for IPv6. You are right, IPv6 networks could have default /64 prefix. However as I wrote in different mail, I don't recommend to use default IPv4 prefix at all because FreeIPA targets for company environments where /24 is not so common, not for home environments. > The installer uses /24 for IPv4 addresses and /64 for IPv6 > addresses, maybe this should be used as a default here as well. Regards, Adam -- Adam Tkac, Red Hat, Inc. From mkosek at redhat.com Wed Sep 5 11:18:06 2012 From: mkosek at redhat.com (Martin Kosek) Date: Wed, 05 Sep 2012 13:18:06 +0200 Subject: [Freeipa-devel] [PATCH] 302 Stricter IP network validator in dnszone-add command In-Reply-To: <20120905110635.GC2549@redhat.com> References: <5047077E.1040000@redhat.com> <50471BBD.1070506@redhat.com> <504727E5.5040101@redhat.com> <50472B1C.2070207@redhat.com> <50472DF6.8000207@redhat.com> <5047314B.30201@redhat.com> <20120905110635.GC2549@redhat.com> Message-ID: <504734EE.1090301@redhat.com> On 09/05/2012 01:06 PM, Adam Tkac wrote: > On Wed, Sep 05, 2012 at 01:02:35PM +0200, Jan Cholasta wrote: >> Dne 5.9.2012 12:48, Martin Kosek napsal(a): >>> On 09/05/2012 12:36 PM, Jan Cholasta wrote: >>>> Dne 5.9.2012 12:22, Petr Spacek napsal(a): >>>>> On 09/05/2012 11:30 AM, Jan Cholasta wrote: >>>>>> Dne 5.9.2012 10:04, Martin Kosek napsal(a): >>>>>>> We allowed IP addresses without network specification which lead >>>>>>> to unexpected results when the zone was being created. We should rather >>>>>>> strictly require the prefix/netmask specifying the IP network that >>>>>>> the reverse zone should be created for. This is already done in >>>>>>> Web UI. >>>>>>> >>>>>>> A unit test exercising this new validation was added. >>>>>>> >>>>>>> https://fedorahosted.org/freeipa/ticket/2461 >>>>>>> >>>>>> >>>>>> I don't like this much. I would suggest using CheckedIPAddress and not >>>>>> forcing >>>>>> the user to enter the prefix length instead. >>>>>> >>>>>> CheckedIPAddress uses a sensible default prefix length if one is not >>>>>> specified >>>>>> (class-based for IPv4, /64 for IPv6) as opposed to IPNetwork (/32 for >>>>>> IPv4, >>>>>> /128 for IPv6 - this causes the erroneous reverse zones to be created as >>>>>> described in the ticket). >>>>>> >>>>> Hello, >>>>> >>>>> I don't like automatic netmask guessing. I have met class-based guessing >>>>> in Windows (XP?) and I was forced to overwrite default mask all the time >>>>> ... >>>> >>>> If there was no guessing, you would have to write the netmask anyway, so I >>>> don't see any harm in guessing here. >>>> >>>>> >>>>> IMHO there is no "sensible default prefix" in real world. I sitting on >>>>> network with /23 prefix right now. Also, I have never seen 10.x network >>>>> with /8 prefix. >>>>> >>>> >>>> While this might be true for IPv4 in some cases, /64 is perfectly sensible for >>>> IPv6. Also, I have never seen 192.168.x.x network with non-/24 prefix. >>>> >>>> Honza >>>> >>> >>> While this may be true for 192.168.x.x, it does not apply for 10.x.x.x networks >>> as Petr already pointed out. I don't think that there will be many people >>> expecting that a reverse zone of 10.0.0.0/24 would be created. >> >> And they would be correct, because the default prefix length for a >> class A network is /8, not /24. >> >>> >>> And since FreeIPA is mainly deployed to internal networks, I assume this will >>> be the case of most users. >>> >>> Martin >>> >> >> OK, but what about IPv6? Correct me if I'm wrong, but the prefix >> length is going to be /64 99% of the time for IPv6. > > You are right, IPv6 networks could have default /64 prefix. However as I wrote > in different mail, I don't recommend to use default IPv4 prefix at all because > FreeIPA targets for company environments where /24 is not so common, not for > home environments. I think now most of us agree on that automatic prefixes does not make much sense for IPv4. But do we really want to introduce different behavior in this case for IPv4+IPv6 network validation and allow automatic prefix only for IPv6 only? > >> The installer uses /24 for IPv4 addresses and /64 for IPv6 >> addresses, maybe this should be used as a default here as well. Yeah, it would make sense to do this change also for installer to make them consistent. Martin From atkac at redhat.com Wed Sep 5 11:29:26 2012 From: atkac at redhat.com (Adam Tkac) Date: Wed, 5 Sep 2012 13:29:26 +0200 Subject: [Freeipa-devel] [PATCH 0051-0052] Log successful reconnection to LDAP server In-Reply-To: <502B85E8.3070302@redhat.com> References: <502B85E8.3070302@redhat.com> Message-ID: <20120905112926.GA9533@redhat.com> On Wed, Aug 15, 2012 at 01:20:08PM +0200, Petr Spacek wrote: > Hello, > > this two patches solves upstream ticket > https://fedorahosted.org/bind-dyndb-ldap/ticket/71 > "Log successful reconnect" > > Patch 51: > Adds log_info(): logging facility with log level INFO. Ack. > Patch 52: > Logs successful reconnection to LDAP server. > > LDAP connection error handling was modified: > Errors are handled exclusively by handle_connection_error() now. > > Direct calls to ldap_connect() and ldap_reconnect() should be avoided. Nack, please check my comments below. Regards, Adam > From 15286f0793d3666845e6b03b565d49f135b115ff Mon Sep 17 00:00:00 2001 > From: Petr Spacek > Date: Wed, 15 Aug 2012 12:05:56 +0200 > Subject: [PATCH] Add log_info(): logging facility with log level INFO. > > Signed-off-by: Petr Spacek > --- > src/log.h | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/src/log.h b/src/log.h > index 898639be144dbf6049a1440493c3358e01a5c2dd..9062a4e80786b5bab806d6c9ceba1d1a9917a3e2 100644 > --- a/src/log.h > +++ b/src/log.h > @@ -43,6 +43,9 @@ > #define log_error(format, ...) \ > log_write(GET_LOG_LEVEL(ISC_LOG_ERROR), format, ##__VA_ARGS__) > > +#define log_info(format, ...) \ > + log_write(GET_LOG_LEVEL(ISC_LOG_INFO), format, ##__VA_ARGS__) > + > #define log_debug(level, format, ...) \ > log_write(GET_LOG_LEVEL(level), format, ##__VA_ARGS__) > > -- > 1.7.11.2 > > From 06f03d8b602656dc9581abc675c943d6fa6a6db2 Mon Sep 17 00:00:00 2001 > From: Petr Spacek > Date: Wed, 15 Aug 2012 12:57:32 +0200 > Subject: [PATCH] Log successful reconnection to LDAP server. > > LDAP connection error handling was modified: > Errors are handled exclusively by handle_connection_error() now. > > Direct calls to ldap_connect() and ldap_reconnect() should be avoided. > > https://fedorahosted.org/bind-dyndb-ldap/ticket/71 > > Signed-off-by: Petr Spacek > --- > src/ldap_helper.c | 37 ++++++++++++++++++++++++------------- > 1 file changed, 24 insertions(+), 13 deletions(-) > > diff --git a/src/ldap_helper.c b/src/ldap_helper.c > index cc7003a6cdcd2d27404fec936623ed8a3e8fa7f8..798aeadfef27d7071a1dd4133b7f08a21918ef78 100644 > --- a/src/ldap_helper.c > +++ b/src/ldap_helper.c > @@ -1734,7 +1734,7 @@ ldap_query(ldap_instance_t *ldap_inst, ldap_connection_t *ldap_conn, > * successful > * TODO: handle this case inside ldap_pool_getconnection()? > */ > - CHECK(ldap_connect(ldap_inst, ldap_conn, ISC_FALSE)); > + CHECK(handle_connection_error(ldap_inst, ldap_conn, ISC_FALSE)); > } > > retry: > @@ -1903,14 +1903,16 @@ ldap_connect(ldap_instance_t *ldap_inst, ldap_connection_t *ldap_conn, > int ret; > int version; > struct timeval timeout; > + isc_result_t result; I would recommend to be consistent here and use "isc_result_t result = ISC_R_FAILURE" > + REQUIRE(ldap_inst != NULL); > REQUIRE(ldap_conn != NULL); > > ret = ldap_initialize(&ld, str_buf(ldap_inst->uri)); > if (ret != LDAP_SUCCESS) { > log_error("LDAP initialization failed: %s", > ldap_err2string(ret)); > - goto cleanup; > + CHECK(ISC_R_FAILURE); Please use goto here, as done in rest of code. > } > > version = LDAP_VERSION3; > @@ -1932,21 +1934,22 @@ ldap_connect(ldap_instance_t *ldap_inst, ldap_connection_t *ldap_conn, > if (ldap_conn->handle != NULL) > ldap_unbind_ext_s(ldap_conn->handle, NULL, NULL); > ldap_conn->handle = ld; > + ld = NULL; /* prevent double-unbind from ldap_reconnect() and cleanup: */ > > - return ldap_reconnect(ldap_inst, ldap_conn, force); > + CHECK(ldap_reconnect(ldap_inst, ldap_conn, force)); > + return result; > > cleanup: > - > if (ld != NULL) > ldap_unbind_ext_s(ld, NULL, NULL); > > /* Make sure handle is NULL. */ > if (ldap_conn->handle != NULL) { > ldap_unbind_ext_s(ldap_conn->handle, NULL, NULL); > ldap_conn->handle = NULL; > } > > - return ISC_R_FAILURE; > + return result; > } > > static isc_result_t > @@ -2064,12 +2067,18 @@ handle_connection_error(ldap_instance_t *ldap_inst, ldap_connection_t *ldap_conn > { > int ret; > int err_code; > + isc_result_t result = ISC_R_FAILURE; > > - ret = ldap_get_option(ldap_conn->handle, LDAP_OPT_RESULT_CODE, > - (void *)&err_code); > + REQUIRE(ldap_conn != NULL); ... > - if (ret != LDAP_OPT_SUCCESS) { > - log_error("handle_connection_error failed to obtain ldap error code"); > + if (ldap_conn->handle != NULL) { > + ret = ldap_get_option(ldap_conn->handle, LDAP_OPT_RESULT_CODE, > + (void *)&err_code); > + if (ret != LDAP_OPT_SUCCESS) { > + log_error("handle_connection_error failed to obtain ldap error code"); > + } > + } > + if (ldap_conn->handle == NULL || ret != LDAP_OPT_SUCCESS) { > goto reconnect; > } I think this is more readable: if (ldap_conn->handle == NULL) goto reconnect; ret = ldap_get_option(..); if (ret != LDAP_OPT_SUCCESS) { log_error goto reconnect; } isn't it? > > @@ -2090,11 +2099,13 @@ handle_connection_error(ldap_instance_t *ldap_inst, ldap_connection_t *ldap_conn > reconnect: > if (ldap_conn->tries == 0) > log_error("connection to the LDAP server was lost"); > - return ldap_connect(ldap_inst, ldap_conn, force); > - > + result = ldap_connect(ldap_inst, ldap_conn, force); > + if (result == ISC_R_SUCCESS) > + log_info("successfully reconnected to LDAP server"); > + break; > } > > - return ISC_R_FAILURE; > + return result; > } > > /* FIXME: Handle the case where the LDAP handle is NULL -> try to reconnect. */ > @@ -3476,7 +3487,7 @@ ldap_psearch_watcher(isc_threadarg_t arg) > "Next try in %ds", inst->reconnect_interval); > if (!sane_sleep(inst, inst->reconnect_interval)) > goto cleanup; > - ldap_connect(inst, conn, ISC_TRUE); > + handle_connection_error(inst, conn, ISC_TRUE); > } > > CHECK(ldap_query_create(conn->mctx, &ldap_qresult)); > -- > 1.7.11.2 > -- Adam Tkac, Red Hat, Inc. From mkosek at redhat.com Wed Sep 5 11:31:46 2012 From: mkosek at redhat.com (Martin Kosek) Date: Wed, 05 Sep 2012 13:31:46 +0200 Subject: [Freeipa-devel] [PATCH] 300-301 Fix DNS SOA serial parameters boundaries In-Reply-To: <504728D9.7020807@redhat.com> References: <50460EB8.2080800@redhat.com> <504725F9.3030509@redhat.com> <504728D9.7020807@redhat.com> Message-ID: <50473822.20702@redhat.com> On 09/05/2012 12:26 PM, Petr Viktorin wrote: > On 09/05/2012 12:14 PM, Petr Viktorin wrote: >> This works well, but please see some comments below. >> >> On 09/04/2012 04:22 PM, Martin Kosek wrote: >>> To test, simply run the following command: >>> >>> ipa dnszone-mod example.com --serial=4294967295 >>> >>> This should work well on patched server&client. Web UI should work too >>> as it >>> reads the max limit dynamically. >> >> Please put this in the test suite. Done. >> >>> --- >>> [PATCH 2/2] Fix DNS SOA serial parameters boundaries: >>> >>> Set correct boundaries for DNS SOA serial parameters (see RFC 1035, >>> 2181). >>> >>> >>> [PATCH 1/2] Transfer long numbers over XMLRPC >>> >>> Numeric parameters in ipalib were limited by XMLRPC boundaries for >>> integer (2^31-1) which is too low for some LDAP attributes like DNS >>> SOA serial field. >>> >>> Transfer numbers which are not in XMLRPC boundary as a string and not >>> as a number to workaround this limitation. Int parameter had to be >>> updated to also accept Python's long type as valid int type. >>> >>> >>> freeipa-mkosek-300-transfer-long-numbers-over-xmlrpc.patch >>> >>> >>>> From 8782015a17b130c5ebae8b014a7241810b10dedd Mon Sep 17 00:00:00 2001 >>> From: Martin Kosek >>> Date: Tue, 4 Sep 2012 15:49:26 +0200 >>> Subject: [PATCH 1/2] Transfer long numbers over XMLRPC >>> >>> Numeric parameters in ipalib were limited by XMLRPC boundaries for >>> integer (2^31-1) which is too low for some LDAP attributes like DNS >>> SOA serial field. >>> >>> Transfer numbers which are not in XMLRPC boundary as a string and not >>> as a number to workaround this limitation. Int parameter had to be >>> updated to also accept Python's long type as valid int type. >>> >>> https://fedorahosted.org/freeipa/ticket/2568 >>> --- >>> ipalib/parameters.py | 12 ++++++------ >>> ipalib/rpc.py | 5 ++++- >>> 2 files changed, 10 insertions(+), 7 deletions(-) >>> >>> diff --git a/ipalib/parameters.py b/ipalib/parameters.py >>> index >>> de0d14faf08d1ab79c99e65dab9cc08f406e3a1d..21e30356b2a351bf7a3be7d47d7fabf0130cf6d4 >>> >>> 100644 >>> --- a/ipalib/parameters.py >>> +++ b/ipalib/parameters.py >>> @@ -1077,7 +1077,7 @@ class Number(Param): >>> """ >>> if type(value) is self.type: >>> return value >>> - if type(value) in (unicode, int, float): >>> + if type(value) in (unicode, int, long, float): >> >> >> PEP8 says that "Object type comparisons should always use isinstance() >> instead of comparing types directly". >> It would be nice to change the old code whenever it is touched. It's >> also in a few more places in the patch. >> I considered doing this when I was developing the patch, but I did not want to mix type/isinstance in the same code. Now, when I experimented and tried to replace it in a larger scope, there were unexpected issues. Like bool type suddenly passing an isinstance(value, (int, long)) test and causing issues later. So I skipped this part (as we discussed off-list). >>> diff --git a/ipalib/rpc.py b/ipalib/rpc.py >>> index >>> d1764e3e30492d5855450398e86689bfcad7aa39..85239ac65903acf447a4d971cce70f819979ce8d >>> >>> 100644 >>> --- a/ipalib/rpc.py >>> +++ b/ipalib/rpc.py >>> @@ -94,6 +95,8 @@ def xml_wrap(value): >>> if type(value) is Decimal: >>> # transfer Decimal as a string >>> return unicode(value) >>> + if isinstance(value, (int, long)) and (value < MININT or value > >>> MAXINT): >>> + return unicode(value) >>> if isinstance(value, DN): >>> return str(value) >>> assert type(value) in (unicode, int, float, bool, NoneType) >> >> `long` should also be included here. >> >> >>> api.Command.user_find(uidnumber=1511000000) >> {'count': 1, 'truncated': False, 'result': ({...},), 'summary': u'1 user >> matched'} >> >>> api.Command.user_find(uidnumber=1511000000 + 0L) >> Traceback (most recent call last): >> File "", line 1, in >> ... >> File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 102, in >> xml_wrap >> assert type(value) in (unicode, int, float, bool, NoneType) >> AssertionError >> >> >> >> > > One more thing: please update the VERSION file. > I did not want to do this because I just messed with validation, but yeah, I can do that. Fixed patches attached. Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-300-2-transfer-long-numbers-over-xmlrpc.patch Type: text/x-patch Size: 3688 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-301-2-fix-dns-soa-serial-parameters-boundaries.patch Type: text/x-patch Size: 12761 bytes Desc: not available URL: From atkac at redhat.com Wed Sep 5 11:33:01 2012 From: atkac at redhat.com (Adam Tkac) Date: Wed, 5 Sep 2012 13:33:01 +0200 Subject: [Freeipa-devel] [PATCH 0053] Use richer set of return codes for LDAP connection error handling code In-Reply-To: <502B86C1.1040907@redhat.com> References: <502B86C1.1040907@redhat.com> Message-ID: <20120905113300.GB9533@redhat.com> On Wed, Aug 15, 2012 at 01:23:45PM +0200, Petr Spacek wrote: > Hello, > > current code return very generic ISC_R_FAILURE code in nearly all (error) cases. > > This patch distinguishes between different LDAP errors and returns > richer set of return codes from LDAP connection error handling code. > > It should lead to clearer log messages. > > Petr^2 Spacek Ack > From 15d6b38c9eda5b05d799c145ede8341f359e8633 Mon Sep 17 00:00:00 2001 > From: Petr Spacek > Date: Wed, 15 Aug 2012 13:01:48 +0200 > Subject: [PATCH] Use richer set of return codes for LDAP connection error > handling code. > > It should lead to clear log messages. > > Signed-off-by: Petr Spacek > --- > src/ldap_helper.c | 9 +++++++-- > 1 file changed, 7 insertions(+), 2 deletions(-) > > diff --git a/src/ldap_helper.c b/src/ldap_helper.c > index 798aeadfef27d7071a1dd4133b7f08a21918ef78..da083d2e65032e650cfbbeb863262e0141403407 100644 > --- a/src/ldap_helper.c > +++ b/src/ldap_helper.c > @@ -1971,7 +1971,7 @@ ldap_reconnect(ldap_instance_t *ldap_inst, ldap_connection_t *ldap_conn, > result = isc_time_now(&now); > time_cmp = isc_time_compare(&now, &ldap_conn->next_reconnect); > if (result == ISC_R_SUCCESS && time_cmp < 0) > - return ISC_R_FAILURE; > + return ISC_R_SOFTQUOTA; > } > > /* If either bind_dn or the password is not set, we will use > @@ -2050,6 +2050,8 @@ force_reconnect: > return ISC_R_NOPERM; > case LDAP_SERVER_DOWN: > return ISC_R_NOTCONNECTED; > + case LDAP_TIMEOUT: > + return ISC_R_TIMEDOUT; > default: > return ISC_R_FAILURE; > } > @@ -2085,13 +2087,16 @@ handle_connection_error(ldap_instance_t *ldap_inst, ldap_connection_t *ldap_conn > switch (err_code) { > case LDAP_NO_SUCH_OBJECT: > ldap_conn->tries = 0; > - return ISC_R_SUCCESS; > + result = ISC_R_SUCCESS; > + break; > case LDAP_TIMEOUT: > log_error("LDAP query timed out. Try to adjust \"timeout\" parameter"); > + result = ISC_R_TIMEDOUT; > break; > case LDAP_INVALID_DN_SYNTAX: > case LDAP_INVALID_SYNTAX: > log_bug("Invalid syntax in handle_connection_error indicates a bug"); > + result = ISC_R_UNEXPECTEDTOKEN; > break; > default: > /* Try to reconnect on other errors. */ > -- > 1.7.11.2 > -- Adam Tkac, Red Hat, Inc. From pspacek at redhat.com Wed Sep 5 11:52:14 2012 From: pspacek at redhat.com (Petr Spacek) Date: Wed, 05 Sep 2012 13:52:14 +0200 Subject: [Freeipa-devel] [PATCH 0050] Fix memory leak in configuration with multiple LDAP instances In-Reply-To: <20120905110222.GB2549@redhat.com> References: <502A59F5.90300@redhat.com> <20120905110222.GB2549@redhat.com> Message-ID: <50473CEE.5080009@redhat.com> On 09/05/2012 01:02 PM, Adam Tkac wrote: > On Tue, Aug 14, 2012 at 04:00:21PM +0200, Petr Spacek wrote: >> Hello, >> >> this patch fixes $SUBJ$. >> >> Adam, please double-check correctness of this change. >> >> I had two assumptions: >> - all locking is done inside dns_db_(un)register() functions >> - LDAP instances are decommissioned before dynamic_driver_destroy() call > > Ack Pushed to master: https://fedorahosted.org/bind-dyndb-ldap/changeset/f390c944336657c604941278b9647449b69c9a8c Petr^2 Spacek From pspacek at redhat.com Wed Sep 5 11:53:31 2012 From: pspacek at redhat.com (Petr Spacek) Date: Wed, 05 Sep 2012 13:53:31 +0200 Subject: [Freeipa-devel] [PATCH] 302 Stricter IP network validator in dnszone-add command In-Reply-To: <504734EE.1090301@redhat.com> References: <5047077E.1040000@redhat.com> <50471BBD.1070506@redhat.com> <504727E5.5040101@redhat.com> <50472B1C.2070207@redhat.com> <50472DF6.8000207@redhat.com> <5047314B.30201@redhat.com> <20120905110635.GC2549@redhat.com> <504734EE.1090301@redhat.com> Message-ID: <50473D3B.9010800@redhat.com> On 09/05/2012 01:18 PM, Martin Kosek wrote: >> >You are right, IPv6 networks could have default /64 prefix. However as I wrote >> >in different mail, I don't recommend to use default IPv4 prefix at all because >> >FreeIPA targets for company environments where /24 is not so common, not for >> >home environments. +1 > I think now most of us agree on that automatic prefixes does not make much > sense for IPv4. But do we really want to introduce different behavior in this > case for IPv4+IPv6 network validation and allow automatic prefix only for IPv6 > only? +1 Petr^2 Spacek From mkosek at redhat.com Wed Sep 5 11:56:15 2012 From: mkosek at redhat.com (Martin Kosek) Date: Wed, 05 Sep 2012 13:56:15 +0200 Subject: [Freeipa-devel] [PATCH 0011] Make sure selinuxusemap behaves consistently to HBAC rule In-Reply-To: <5044C8CE.20108@redhat.com> References: <5044C8CE.20108@redhat.com> Message-ID: <50473DDF.8040303@redhat.com> On 09/03/2012 05:12 PM, Tomas Babej wrote: > Hi, > > Both selinuxusermap-add and selinuxusermap-mod commands now behave > consistently in not allowing user/host category or user/host members > and HBAC rule being set at the same time. Also adds a bunch of unit > tests that check this behaviour. > > https://fedorahosted.org/freeipa/ticket/2983 > > Tomas > I found few issues with this patch: 1) Patch needs a rebase 2) Patch does not expect attributes to be set to None, i.e. to be left empty or to be deleted, e.g.: # ipa selinuxusermap-add foo --selinuxuser=guest_u:s0 --usercat=all --hbacrule= ipa: ERROR: HBAC rule and local members cannot both be set # ipa selinuxusermap-add foo --selinuxuser=guest_u:s0 --usercat=all ---------------------------- Added SELinux User Map "foo" ---------------------------- Rule name: foo SELinux User: guest_u:s0 User category: all Enabled: TRUE # ipa selinuxusermap-mod foo --usercat= --hbacrule= ipa: ERROR: HBAC rule and local members cannot both be set # ipa selinuxusermap-mod foo --usercat= ------------------------------- Modified SELinux User Map "foo" ------------------------------- Rule name: foo SELinux User: guest_u:s0 Enabled: TRUE # ipa selinuxusermap-mod foo --hbacrule=foo ------------------------------- Modified SELinux User Map "foo" ------------------------------- Rule name: foo SELinux User: guest_u:s0 HBAC Rule: foo Enabled: TRUE # ipa selinuxusermap-mod foo --hbacrule= --usercat=all ipa: ERROR: HBAC rule and local members cannot both be set All these validation failures are not valid. 3) Additionally, I think it would be more readable and less error prone that if instead of this blob: + are_local_members_to_be_set = 'usercategory' in _entry_attrs or \ + 'hostcategory' in _entry_attrs or \ + 'memberuser' in _entry_attrs or \ + 'memberhost' in _entry_attrs You would use something like that: are_local_members_to_be_set = any(attr in _entry_attrs for attr in ('usercategory', 'hostcategory', 'memberuser', 'memberhost')) Martin From simo at redhat.com Wed Sep 5 12:26:19 2012 From: simo at redhat.com (Simo Sorce) Date: Wed, 05 Sep 2012 08:26:19 -0400 Subject: [Freeipa-devel] [PATCH] 302 Stricter IP network validator in dnszone-add command In-Reply-To: <50471BBD.1070506@redhat.com> References: <5047077E.1040000@redhat.com> <50471BBD.1070506@redhat.com> Message-ID: <1346847979.1150.21.camel@willson.li.ssimo.org> On Wed, 2012-09-05 at 11:30 +0200, Jan Cholasta wrote: > Dne 5.9.2012 10:04, Martin Kosek napsal(a): > > We allowed IP addresses without network specification which lead > > to unexpected results when the zone was being created. We should rather > > strictly require the prefix/netmask specifying the IP network that > > the reverse zone should be created for. This is already done in > > Web UI. > > > > A unit test exercising this new validation was added. > > > > https://fedorahosted.org/freeipa/ticket/2461 > > > > I don't like this much. I would suggest using CheckedIPAddress and not > forcing the user to enter the prefix length instead. > > CheckedIPAddress uses a sensible default prefix length if one is not > specified (class-based for IPv4, /64 for IPv6) IPv4 classes were already dead and not relevant last century Jan, so class-based netmask is really useless, if we want to use a default for ipv4 I would use /24 for any address, that's the simplest guess you can make ,a nd will still be often wrongt, but certainly less wrong than using the outdated 'class' concept. Simo. > as opposed to IPNetwork > (/32 for IPv4, /128 for IPv6 - this causes the erroneous reverse zones > to be created as described in the ticket). > > Honza > -- Simo Sorce * Red Hat, Inc * New York From mkosek at redhat.com Wed Sep 5 12:25:12 2012 From: mkosek at redhat.com (Martin Kosek) Date: Wed, 05 Sep 2012 14:25:12 +0200 Subject: [Freeipa-devel] [PATCH] ipadb_iterate(): handle match_entry == NULL In-Reply-To: <1292598576.8371089.1345554397961.JavaMail.root@redhat.com> References: <1292598576.8371089.1345554397961.JavaMail.root@redhat.com> Message-ID: <504744A8.6080800@redhat.com> On 08/21/2012 03:06 PM, Simo Sorce wrote: > ----- Original Message ----- >> ----- Original Message ----- >>> Hi, >>> >>> there was an issue reported yesterday on #freeipa >>> (https://fedorahosted.org/freeipa/ticket/3011). It is easy to >>> reproduce >>> 'kdb5_util dump' just core dumps. The attached patch adds a >>> parameter >>> check to the call where the core dump occured and fixes the reason >>> why >>> the parameter was invalid. >>> >>> Please note that 'kdb5_util dump' will return 'kdb5_util: error >>> performing Kerberos version 5 release 1.8 dump (Plugin does not >>> support >>> the operation)' with the patch applied, because >>> ipadb_iterate_pwd_policy() is not implemented. >> >> Can you open a ticket on this ? We used the dump utility in the past >> to perform bulk master key changes in the past and we do want to >> have the option open bot for backup purposes and other reasons. > > Also obvious ack to the patch. > > Simo. > Verified that this patch removes the segfault. ACK#2. Pushed to master, ipa-3-0. Martin From rcritten at redhat.com Wed Sep 5 12:26:26 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 05 Sep 2012 08:26:26 -0400 Subject: [Freeipa-devel] [PATCH] 0077 Check direct/reverse hostname/address resolution in ipa-replica-install In-Reply-To: <5046FF0E.1000504@redhat.com> References: <5046009E.2020306@redhat.com> <50463DE6.9070300@redhat.com> <5046FF0E.1000504@redhat.com> Message-ID: <504744F2.4060701@redhat.com> Petr Viktorin wrote: > On 09/04/2012 07:44 PM, Rob Crittenden wrote: >> Petr Viktorin wrote: >>> >>> https://fedorahosted.org/freeipa/ticket/2845 >> >> Shouldn't this also call verify_fqdn() on the local hostname and not >> just the master? I think this would eventually fail in the conncheck but >> what if that was skipped? >> >> rob > > A few lines above there is a call to get_host_name, which will call > verify_fqdn. > Ok, well I tested this by commenting out the host's entry in /etc/hosts and removing it from DNS, and it failed at the conn check. I suppose I could have nscd still running, I'll take another look. rob From mkosek at redhat.com Wed Sep 5 12:47:00 2012 From: mkosek at redhat.com (Martin Kosek) Date: Wed, 05 Sep 2012 14:47:00 +0200 Subject: [Freeipa-devel] [PATCH] 298 Add safe updates for objectClasses In-Reply-To: <5046FDD3.9060200@redhat.com> References: <503E0DD9.2020905@redhat.com> <503F6249.3060209@redhat.com> <5045EED8.70005@redhat.com> <50467742.5020900@redhat.com> <5046AF40.40408@redhat.com> <5046FDD3.9060200@redhat.com> Message-ID: <504749C4.1090108@redhat.com> On 09/05/2012 09:22 AM, Martin Kosek wrote: > On 09/05/2012 03:47 AM, Rob Crittenden wrote: >> Rob Crittenden wrote: >>> Martin Kosek wrote: >>>> On 08/30/2012 02:53 PM, Rob Crittenden wrote: >>>>> Martin Kosek wrote: >>>>>> Current objectclass updates in a form of "replace" update instruction >>>>>> dependent on exact match of the old object class specification in the >>>>>> update instruction and the real value in LDAP. However, this >>>>>> approach is >>>>>> very error prone as object class definition can easily differ as for >>>>>> example because of unexpected X-ORIGIN value. Such objectclass update >>>>>> failures may lead to serious malfunctions later. >>>>>> >>>>>> Add new update instruction type "replaceoc" with the following format: >>>>>> replaceoc:OID:new >>>>>> This update instruction will always replace an objectclass with >>>>>> specified OID with the new definition. >>>>>> >>>>>> https://fedorahosted.org/freeipa/ticket/2440 >>>>> >>>>> This works ok. Martin and I had a conversation in IRC about it. >>>>> >>>>> This moves from replacing a specific bit of schema with a new one, in >>>>> all >>>>> cases. I wonder if we should be more conservative and know what we're >>>>> replacing >>>>> in advance. >>>>> >>>>> rob >>>>> >>>> >>>> You are right, I was too harsh when replacing the objectclasses. This >>>> would >>>> cause issues when LDAP update would be run on a replica with lower >>>> version and >>>> older objectclass definitions. >>>> >>>> I came up with an alternative solution and instead of always replacing >>>> the >>>> objectclass I rather reverted to old-OC:new-OC style which should be >>>> safer. >>>> Now, the LDAP updater always normalizes an objectclass before >>>> comparing it >>>> using python-ldap objectclass model. With this approach, objectclasses >>>> differing only in X-ORIGIN or white spaces should match and be updated. >>>> >>>> Martin >>>> >>> >>> NACK >>> >>> I think this: >>> >>> + for value in replaced_values: >>> + entry_values.remove(old) >>> >>> Should be: >>> >>> + entry_values.remove(value) > > Right. Thanks for the fix. It only worked in my testing because I had no > objectclass update which would differ in X-ORIGIN or whitespaces or case. I > tried to mangle an update file and with this fix it did the update even if > X-ORIGIN and whitespaces differed. > >>> >>> I'm still doing other testing but this is what I've found so far. >>> >>> rob >> >> I did some more testing and it looks like this will do the trick. >> >> I also found a place where the schema was left as unicode and causing it to >> blow up inside python-ldap. Here is the diff on my working instance: >> >> diff -u ipaserver/install/ldapupdate.py >> /usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py >> --- ipaserver/install/ldapupdate.py 2012-09-04 16:59:33.210688723 -0400 >> +++ /usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py 2012-09-04 >> 21:47:01.583574375 -0400 >> @@ -643,7 +643,7 @@ >> self.debug('replace: no match for replaced >> ObjectClass "%s"', old) >> continue >> for value in replaced_values: >> - entry_values.remove(old) >> + entry_values.remove(value) >> else: >> entry_values.remove(old) >> entry_values.append(new) >> @@ -772,7 +772,11 @@ >> updated = False >> changes = self.conn.generateModList(entry.origDataDict(), >> entry.toDict()) >> if (entry.dn == DN(('cn', 'schema'))): >> - updated = self.is_schema_updated(entry.toDict()) >> + d = dict() >> + e = entry.toDict() >> + for k,v in e.items(): >> + d[k] = [str(x) for x in v] >> + updated = self.is_schema_updated(d) >> else: >> if len(changes) >= 1: >> updated = True >> >> rob >> > > I did not hit this error during testing, but at least it won't harm. Sending an > updated patch. > Sending a slightly updated patch which now makes sure that we always pass objectclass of str (and not unicode) type to ObjectClass model. This should make it more bullet-proof. Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-298-4-add-safe-updates-for-objectclasses.patch Type: text/x-patch Size: 6768 bytes Desc: not available URL: From tbabej at redhat.com Wed Sep 5 13:19:12 2012 From: tbabej at redhat.com (Tomas Babej) Date: Wed, 05 Sep 2012 15:19:12 +0200 Subject: [Freeipa-devel] [PATCH 0013] Remove user-unfriendly "u" character from error messages Message-ID: <50475150.4090905@redhat.com> Hi, User-unfriendly errors were caused by re-raising errors from external python module netaddr. https://fedorahosted.org/freeipa/ticket/2588 Tomas -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-tbabej-0013-Remove-user-unfriendly-u-character-from-error-messag.patch Type: text/x-patch Size: 1935 bytes Desc: not available URL: From pviktori at redhat.com Wed Sep 5 13:42:07 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Wed, 05 Sep 2012 15:42:07 +0200 Subject: [Freeipa-devel] [PATCH 0013] Remove user-unfriendly "u" character from error messages In-Reply-To: <50475150.4090905@redhat.com> References: <50475150.4090905@redhat.com> Message-ID: <504756AF.1050803@redhat.com> On 09/05/2012 03:19 PM, Tomas Babej wrote: > Hi, > > User-unfriendly errors were caused by re-raising errors > from external python module netaddr. > > https://fedorahosted.org/freeipa/ticket/2588 > > Tomas > > > _______________________________________________ > Freeipa-devel mailing list > Freeipa-devel at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-devel > I don't agree with this approach. Raising another module's errors in our code is ugly, and will break if netaddr changes. The arguments to pass to the exceptions are undocumented (see http://packages.python.org/netaddr/api.html#custom-exceptions). The wording of error messages in libraries can usually change at any time, and is intended for developers, not end users. This should be either fixed upstream (unlikely, using the repr() of the argument is a sane thing to do at their side), or we should pass bytestrings to netaddr (a possible quick fix, not sure it it'll work), or, ideally, we should raise IPA's own errors. -- Petr? From pspacek at redhat.com Wed Sep 5 13:53:36 2012 From: pspacek at redhat.com (Petr Spacek) Date: Wed, 05 Sep 2012 15:53:36 +0200 Subject: [Freeipa-devel] [PATCH 0051-0052] Log successful reconnection to LDAP server In-Reply-To: <20120905112926.GA9533@redhat.com> References: <502B85E8.3070302@redhat.com> <20120905112926.GA9533@redhat.com> Message-ID: <50475960.10600@redhat.com> On 09/05/2012 01:29 PM, Adam Tkac wrote: > On Wed, Aug 15, 2012 at 01:20:08PM +0200, Petr Spacek wrote: >> Hello, >> >> this two patches solves upstream ticket >> https://fedorahosted.org/bind-dyndb-ldap/ticket/71 >> "Log successful reconnect" >> >> Patch 51: >> Adds log_info(): logging facility with log level INFO. > > Ack. > >> Patch 52: >> Logs successful reconnection to LDAP server. >> >> LDAP connection error handling was modified: >> Errors are handled exclusively by handle_connection_error() now. >> >> Direct calls to ldap_connect() and ldap_reconnect() should be avoided. > > Nack, please check my comments below. Thanks for your comments! Corrected patches are attached + I replied in-line. > > Regards, Adam > > >> From 06f03d8b602656dc9581abc675c943d6fa6a6db2 Mon Sep 17 00:00:00 2001 >> From: Petr Spacek >> Date: Wed, 15 Aug 2012 12:57:32 +0200 >> Subject: [PATCH] Log successful reconnection to LDAP server. >> >> LDAP connection error handling was modified: >> Errors are handled exclusively by handle_connection_error() now. >> >> Direct calls to ldap_connect() and ldap_reconnect() should be avoided. >> >> https://fedorahosted.org/bind-dyndb-ldap/ticket/71 >> >> Signed-off-by: Petr Spacek >> --- >> src/ldap_helper.c | 37 ++++++++++++++++++++++++------------- >> 1 file changed, 24 insertions(+), 13 deletions(-) >> >> diff --git a/src/ldap_helper.c b/src/ldap_helper.c >> index cc7003a6cdcd2d27404fec936623ed8a3e8fa7f8..798aeadfef27d7071a1dd4133b7f08a21918ef78 100644 >> --- a/src/ldap_helper.c >> +++ b/src/ldap_helper.c >> @@ -1734,7 +1734,7 @@ ldap_query(ldap_instance_t *ldap_inst, ldap_connection_t *ldap_conn, >> * successful >> * TODO: handle this case inside ldap_pool_getconnection()? >> */ >> - CHECK(ldap_connect(ldap_inst, ldap_conn, ISC_FALSE)); >> + CHECK(handle_connection_error(ldap_inst, ldap_conn, ISC_FALSE)); >> } >> >> retry: >> @@ -1903,14 +1903,16 @@ ldap_connect(ldap_instance_t *ldap_inst, ldap_connection_t *ldap_conn, >> int ret; >> int version; >> struct timeval timeout; >> + isc_result_t result; > > I would recommend to be consistent here and use "isc_result_t result = ISC_R_FAILURE" My fault, corrected. It is definitely better to initialize it to avoid surprises in a future. >> + REQUIRE(ldap_inst != NULL); >> REQUIRE(ldap_conn != NULL); >> >> ret = ldap_initialize(&ld, str_buf(ldap_inst->uri)); >> if (ret != LDAP_SUCCESS) { >> log_error("LDAP initialization failed: %s", >> ldap_err2string(ret)); >> - goto cleanup; >> + CHECK(ISC_R_FAILURE); > > Please use goto here, as done in rest of code. My intent was to replace result = ISC_R_...; goto cleanup; with CHECK(ISC_R_...); to make it more readable. I don't like "alone" goto, because "result" value can change after code changes in future. Well, CHECK() wasn't good choice for this. I introduced CLEANUP_WITH() macro to distinguish jump with constant result from CHECK(function_call()) cases. Attached patch 0052a adds CLEANUP_WITH() macro. This distinguishing should allow to realize failure injection to CHECK() macros (in far future :-). >> } >> >> version = LDAP_VERSION3; >> @@ -1932,21 +1934,22 @@ ldap_connect(ldap_instance_t *ldap_inst, ldap_connection_t *ldap_conn, >> if (ldap_conn->handle != NULL) >> ldap_unbind_ext_s(ldap_conn->handle, NULL, NULL); >> ldap_conn->handle = ld; >> + ld = NULL; /* prevent double-unbind from ldap_reconnect() and cleanup: */ >> >> - return ldap_reconnect(ldap_inst, ldap_conn, force); >> + CHECK(ldap_reconnect(ldap_inst, ldap_conn, force)); >> + return result; >> >> cleanup: >> - >> if (ld != NULL) >> ldap_unbind_ext_s(ld, NULL, NULL); >> >> /* Make sure handle is NULL. */ >> if (ldap_conn->handle != NULL) { >> ldap_unbind_ext_s(ldap_conn->handle, NULL, NULL); >> ldap_conn->handle = NULL; >> } >> >> - return ISC_R_FAILURE; >> + return result; >> } >> >> static isc_result_t >> @@ -2064,12 +2067,18 @@ handle_connection_error(ldap_instance_t *ldap_inst, ldap_connection_t *ldap_conn >> { >> int ret; >> int err_code; >> + isc_result_t result = ISC_R_FAILURE; >> >> - ret = ldap_get_option(ldap_conn->handle, LDAP_OPT_RESULT_CODE, >> - (void *)&err_code); >> + REQUIRE(ldap_conn != NULL); > > ... > >> - if (ret != LDAP_OPT_SUCCESS) { >> - log_error("handle_connection_error failed to obtain ldap error code"); >> + if (ldap_conn->handle != NULL) { >> + ret = ldap_get_option(ldap_conn->handle, LDAP_OPT_RESULT_CODE, >> + (void *)&err_code); >> + if (ret != LDAP_OPT_SUCCESS) { >> + log_error("handle_connection_error failed to obtain ldap error code"); >> + } >> + } >> + if (ldap_conn->handle == NULL || ret != LDAP_OPT_SUCCESS) { >> goto reconnect; >> } > > I think this is more readable: > > if (ldap_conn->handle == NULL) > goto reconnect; > > ret = ldap_get_option(..); > if (ret != LDAP_OPT_SUCCESS) { > log_error > goto reconnect; > } > > isn't it? Yes, it is. You saw my unsuccessful attempt to concentrate gotos to one place. I replaced it with you proposal (actually the original code :-)). Petr^2 Spacek > >> >> @@ -2090,11 +2099,13 @@ handle_connection_error(ldap_instance_t *ldap_inst, ldap_connection_t *ldap_conn >> reconnect: >> if (ldap_conn->tries == 0) >> log_error("connection to the LDAP server was lost"); >> - return ldap_connect(ldap_inst, ldap_conn, force); >> - >> + result = ldap_connect(ldap_inst, ldap_conn, force); >> + if (result == ISC_R_SUCCESS) >> + log_info("successfully reconnected to LDAP server"); >> + break; >> } >> >> - return ISC_R_FAILURE; >> + return result; >> } >> >> /* FIXME: Handle the case where the LDAP handle is NULL -> try to reconnect. */ >> @@ -3476,7 +3487,7 @@ ldap_psearch_watcher(isc_threadarg_t arg) >> "Next try in %ds", inst->reconnect_interval); >> if (!sane_sleep(inst, inst->reconnect_interval)) >> goto cleanup; >> - ldap_connect(inst, conn, ISC_TRUE); >> + handle_connection_error(inst, conn, ISC_TRUE); >> } >> >> CHECK(ldap_query_create(conn->mctx, &ldap_qresult)); >> -- >> 1.7.11.2 >> > > -------------- next part -------------- A non-text attachment was scrubbed... Name: bind-dyndb-ldap-pspacek-0052-2-Log-successful-reconnection-to-LDAP-server.patch Type: text/x-patch Size: 3620 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bind-dyndb-ldap-pspacek-0052a-Add-CLEANUP_WITH-macro-for-unconditional-result-code.patch Type: text/x-patch Size: 804 bytes Desc: not available URL: From pvoborni at redhat.com Wed Sep 5 14:08:04 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Wed, 05 Sep 2012 16:08:04 +0200 Subject: [Freeipa-devel] [PATCH] 206-208 Fixed number parameters MIN boundary validation in Web UI Message-ID: <50475CC4.5010205@redhat.com> Integers were missing most of minimum checks and Decimals boundaries weren't checked at all in Web UI. First part is done in ipalib, second in Web UI. 1) [PATCH] 206 Fixed metadata serialization of Numbers and DNs: There were following problems: 1. DNs and Decimals weren't properly serialized. Serialization output was object with empty __base64__ attribute. It was fixed by converting them to string. 2. numberical values equal to 0 were excluded from metadata. It broke many of minvalue checks in Web UI. Now excluding only None and False values as initially intended. 2)[PATCH] 207 Added decimal checks to metadata validator: Metadata validator didn't have check for decimal values. It was added. 3)[PATCH] 208 Generated metadata for testing updated Testing metadata needs to be updated because of fix in json serialization. -- Petr Vobornik -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pvoborni-0206-Fixed-metadata-serialization-of-Numbers-and-DNs.patch Type: text/x-patch Size: 3401 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pvoborni-0207-Added-decimal-checks-to-metadata-validator.patch Type: text/x-patch Size: 3639 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pvoborni-0208-Generated-metadata-for-testing-updated.patch Type: text/x-patch Size: 100620 bytes Desc: not available URL: From pspacek at redhat.com Wed Sep 5 14:17:50 2012 From: pspacek at redhat.com (Petr Spacek) Date: Wed, 05 Sep 2012 16:17:50 +0200 Subject: [Freeipa-devel] [PATCH 0042] Flush zones and RRs cache when handling persistent search reconnection In-Reply-To: <503B6AAF.7030003@redhat.com> References: <190240025.5168295.1345022311962.JavaMail.root@redhat.com> <503B6AAF.7030003@redhat.com> Message-ID: <50475F0E.4050609@redhat.com> On 08/27/2012 02:40 PM, Dmitri Pal wrote: > On 08/15/2012 05:18 AM, Simo Sorce wrote: >> ----- Original Message ----- >>> On 08/14/2012 08:25 PM, Simo Sorce wrote: >>>> See man ldap_result, the entries return with type >>>> LDAP_RES_SEARCH_ENTRY, the last message is instead >>>> LDAP_RES_SEARCH_RESULT which tells you the searc is complete. >>>> >>>> This last message is never sent for a persistent search of course >>>> (IIRC :-). >>> Right, it is what I tried to say with "there is no SearchResultDone >>> LDAP message". >> I see. >> >>> http://tools.ietf.org/html/draft-smith-psearch-ldap-01#page-3 >>> section 4 paragraph b). >>> >>> This patch deals with persistent search only. Non-persistent search >>> scenarios >>> have cache records with limited TTL value. >>> >>>> But in case of a persistent search you should never need to flush >>>> as entries are valid until you see a change. >>> Unfortunately, cache flush is necessary after persistent search >>> reconnection. >>> Records can change in meanwhile... >>> >>> Example: >>> 1. BIND started, cache was populated. >>> 2. Persistent search connection was lost >>> 3. Record was deleted (but Entry Change Notification wasn't >>> delivered) >>> 4. Persistent search connection was re-established - there is no >>> information >>> about deleted record/zone, BIND still sees old data in the cache. >>> >>> For this reason https://fedorahosted.org/bind-dyndb-ldap/ticket/44 >>> was filled. >>> >>> What I missed? >> You missed nothing, I did. >> >> Howeve I have an idea to handle this situation smartly. >> >> We can issue 2 searches on 2 separate connections. >> >> The first search will be a persistent search, however it will be started with a filter that has an additional element. >> Before the persistent search we do a rootdse search and find either the higher modifyTimestamp or the higher entryUSN or equivalent, depending on what is available in the server. Worst case we do a ase search of the DNS tree and pick that modifyTimestamp. >> Then we start the persistent search with (&(entryUSN>%d)) or (&(modifyTimestamp>=%d) (note modifyTimestamp requires >= due to low resolution). >> >> On the other connection we start instead a nornmal search. This normal search will terminate with a 'done' result, so you know that once that search is finished you can flush any cache that has not been re-validated. While you get any changes that are occurring live on the persistent search connection. >> >> This way we should be able to not loose any update and still know when we got all results and drop unvalidated caches. >> > > What is the status of this proposal? Oh, this mail got lost in my inbox, sorry. I created ticket: https://fedorahosted.org/bind-dyndb-ldap/ticket/86 "Improve psearch cache flush reliability" I will return back to this proposal after resolving more actual issues. Petr^2 Spacek From tbabej at redhat.com Wed Sep 5 14:35:03 2012 From: tbabej at redhat.com (Tomas Babej) Date: Wed, 05 Sep 2012 16:35:03 +0200 Subject: [Freeipa-devel] [PATCH 0013] Remove user-unfriendly "u" character from error messages In-Reply-To: <504756AF.1050803@redhat.com> References: <50475150.4090905@redhat.com> <504756AF.1050803@redhat.com> Message-ID: <50476317.20604@redhat.com> On 09/05/2012 03:42 PM, Petr Viktorin wrote: > On 09/05/2012 03:19 PM, Tomas Babej wrote: >> Hi, >> >> User-unfriendly errors were caused by re-raising errors >> from external python module netaddr. >> >> https://fedorahosted.org/freeipa/ticket/2588 >> >> Tomas >> >> >> _______________________________________________ >> Freeipa-devel mailing list >> Freeipa-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/freeipa-devel >> > > I don't agree with this approach. Raising another module's errors in > our code is ugly, and will break if netaddr changes. The arguments to > pass to the exceptions are undocumented (see > http://packages.python.org/netaddr/api.html#custom-exceptions). The > wording of error messages in libraries can usually change at any time, > and is intended for developers, not end users. > > This should be either fixed upstream (unlikely, using the repr() of > the argument is a sane thing to do at their side), or we should pass > bytestrings to netaddr (a possible quick fix, not sure it it'll work), > or, ideally, we should raise IPA's own errors. > Well, this particular fix wouldn't have broken anything, since it was raising the same error that the except clause in which the raising occured caught. However, I changed this to StandardError, since the error message is extracted and packed into ValidationError during further validation and therefore simple format message is suitable. Tomas -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-tbabej-0013-2-Remove-user-unfriendly-u-character-from-error-messag.patch Type: text/x-patch Size: 2141 bytes Desc: not available URL: From atkac at redhat.com Wed Sep 5 14:37:45 2012 From: atkac at redhat.com (Adam Tkac) Date: Wed, 5 Sep 2012 16:37:45 +0200 Subject: [Freeipa-devel] [PATCH 0051-0052] Log successful reconnection to LDAP server In-Reply-To: <50475960.10600@redhat.com> References: <502B85E8.3070302@redhat.com> <20120905112926.GA9533@redhat.com> <50475960.10600@redhat.com> Message-ID: <20120905143745.GA19401@redhat.com> On Wed, Sep 05, 2012 at 03:53:36PM +0200, Petr Spacek wrote: > On 09/05/2012 01:29 PM, Adam Tkac wrote: > >On Wed, Aug 15, 2012 at 01:20:08PM +0200, Petr Spacek wrote: > >>Hello, > >> > >>this two patches solves upstream ticket > >>https://fedorahosted.org/bind-dyndb-ldap/ticket/71 > >>"Log successful reconnect" > >> > >>Patch 51: > >> Adds log_info(): logging facility with log level INFO. > > > >Ack. > > > >>Patch 52: > >> Logs successful reconnection to LDAP server. > >> > >> LDAP connection error handling was modified: > >> Errors are handled exclusively by handle_connection_error() now. > >> > >> Direct calls to ldap_connect() and ldap_reconnect() should be avoided. > > > >Nack, please check my comments below. > > Thanks for your comments! Corrected patches are attached + I replied in-line. Ack -- Adam Tkac, Red Hat, Inc. From pvoborni at redhat.com Wed Sep 5 15:00:45 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Wed, 05 Sep 2012 17:00:45 +0200 Subject: [Freeipa-devel] [PATCH] 209 Fixed problem while deleting entry with unsaved changes Message-ID: <5047691D.6010108@redhat.com> While deleting an entry it now resets a facet if there are unsaved changes. It prevents pop up of various error dialogs when UI tries to redirect to search page after successful delete. https://fedorahosted.org/freeipa/ticket/3047 -- Petr Vobornik -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pvoborni-0209-Fixed-problem-while-deleting-entry-with-unsaved-chan.patch Type: text/x-patch Size: 1879 bytes Desc: not available URL: From mkosek at redhat.com Wed Sep 5 15:13:41 2012 From: mkosek at redhat.com (Martin Kosek) Date: Wed, 05 Sep 2012 17:13:41 +0200 Subject: [Freeipa-devel] [PATCH] 303 Add range safety check for range_mod and range_del Message-ID: <50476C25.9000009@redhat.com> range_mod and range_del command could easily create objects with ID which is suddenly out of specified range. This could cause issues in trust scenarios where range objects are used for computation of remote IDs. Add validator for both commands to check if there is any object with ID in the range which would become out-of-range as a pre_callback. Also add unit tests testing this new validator. https://fedorahosted.org/freeipa/ticket/2919 -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-303-add-range-safety-check-for-range_mod-and-range_del.patch Type: text/x-patch Size: 13743 bytes Desc: not available URL: From pspacek at redhat.com Wed Sep 5 16:10:07 2012 From: pspacek at redhat.com (Petr Spacek) Date: Wed, 05 Sep 2012 18:10:07 +0200 Subject: [Freeipa-devel] [PATCH 0051-0052] Log successful reconnection to LDAP server In-Reply-To: <20120905143745.GA19401@redhat.com> References: <502B85E8.3070302@redhat.com> <20120905112926.GA9533@redhat.com> <50475960.10600@redhat.com> <20120905143745.GA19401@redhat.com> Message-ID: <5047795F.7010902@redhat.com> On 09/05/2012 04:37 PM, Adam Tkac wrote: > On Wed, Sep 05, 2012 at 03:53:36PM +0200, Petr Spacek wrote: >> On 09/05/2012 01:29 PM, Adam Tkac wrote: >>> On Wed, Aug 15, 2012 at 01:20:08PM +0200, Petr Spacek wrote: >>>> Hello, >>>> >>>> this two patches solves upstream ticket >>>> https://fedorahosted.org/bind-dyndb-ldap/ticket/71 >>>> "Log successful reconnect" >>>> >>>> Patch 51: >>>> Adds log_info(): logging facility with log level INFO. >>> >>> Ack. Pushed to master: https://fedorahosted.org/bind-dyndb-ldap/changeset/3d5310d29f709f83fabb33f54356efa2acf5582d >>> >>>> Patch 52: >>>> Logs successful reconnection to LDAP server. >>>> >>>> LDAP connection error handling was modified: >>>> Errors are handled exclusively by handle_connection_error() now. >>>> >>>> Direct calls to ldap_connect() and ldap_reconnect() should be avoided. >>> >>> Nack, please check my comments below. >> >> Thanks for your comments! Corrected patches are attached + I replied in-line. > > Ack > Pushed to master: https://fedorahosted.org/bind-dyndb-ldap/changeset/ea9c3287f9e01dd932116e39de6cb592862f7591 (CLEANUP_WITH macro) and https://fedorahosted.org/bind-dyndb-ldap/e89735d969dc79d632f2c7776f5e88b934ccce57 (patch itself) Petr^2 Spacek From pspacek at redhat.com Wed Sep 5 16:11:11 2012 From: pspacek at redhat.com (Petr Spacek) Date: Wed, 05 Sep 2012 18:11:11 +0200 Subject: [Freeipa-devel] [PATCH 0053] Use richer set of return codes for LDAP connection error handling code In-Reply-To: <20120905113300.GB9533@redhat.com> References: <502B86C1.1040907@redhat.com> <20120905113300.GB9533@redhat.com> Message-ID: <5047799F.4080302@redhat.com> On 09/05/2012 01:33 PM, Adam Tkac wrote: > On Wed, Aug 15, 2012 at 01:23:45PM +0200, Petr Spacek wrote: >> Hello, >> >> current code return very generic ISC_R_FAILURE code in nearly all (error) cases. >> >> This patch distinguishes between different LDAP errors and returns >> richer set of return codes from LDAP connection error handling code. >> >> It should lead to clearer log messages. >> >> Petr^2 Spacek > > Ack Pushed to master: https://fedorahosted.org/bind-dyndb-ldap/changeset/64310eb7c9a5c0489c82f2f31627874bf7c6b230 Petr^2 Spacek > >> From 15d6b38c9eda5b05d799c145ede8341f359e8633 Mon Sep 17 00:00:00 2001 >> From: Petr Spacek >> Date: Wed, 15 Aug 2012 13:01:48 +0200 >> Subject: [PATCH] Use richer set of return codes for LDAP connection error >> handling code. >> >> It should lead to clear log messages. >> From edewata at redhat.com Wed Sep 5 17:24:47 2012 From: edewata at redhat.com (Endi Sukma Dewata) Date: Wed, 05 Sep 2012 12:24:47 -0500 Subject: [Freeipa-devel] [PATCH] 197 Fixed search in HBAC test In-Reply-To: <50449453.2010404@redhat.com> References: <50339523.1070909@redhat.com> <503CD8EC.6050503@redhat.com> <50449453.2010404@redhat.com> Message-ID: <50478ADF.7070100@redhat.com> On 9/3/2012 6:28 AM, Petr Vobornik wrote: >>> b) force refresh when searching with unchanged filter > > I did (b). Updated patch attached. > > I don't want to implement 'expiration date' at the moment. It's too > widespread change. Maybe in FreeIPA 3.2. ACK. -- Endi S. Dewata From edewata at redhat.com Wed Sep 5 17:24:54 2012 From: edewata at redhat.com (Endi Sukma Dewata) Date: Wed, 05 Sep 2012 12:24:54 -0500 Subject: [Freeipa-devel] [PATCH] 199 Permissions: select only applicable options on type change In-Reply-To: <50448D95.2080605@redhat.com> References: <5034CDD9.2050900@redhat.com> <503CD9AA.8090207@redhat.com> <50448D95.2080605@redhat.com> Message-ID: <50478AE6.9090202@redhat.com> On 9/3/2012 5:59 AM, Petr Vobornik wrote: > Updated patch attached. ACK. -- Endi S. Dewata From edewata at redhat.com Wed Sep 5 17:25:48 2012 From: edewata at redhat.com (Endi Sukma Dewata) Date: Wed, 05 Sep 2012 12:25:48 -0500 Subject: [Freeipa-devel] [PATCH] 203 Notify success on add, delete and update In-Reply-To: <5044B22D.7030602@redhat.com> References: <5044B22D.7030602@redhat.com> Message-ID: <50478B1C.8010504@redhat.com> On 9/3/2012 8:35 AM, Petr Vobornik wrote: > Notification of success was added to: > * details facet: update > * association facet and association widget: add, delete items > * attribute facet: delete items (notification of add should be handled > in entity adder dialog) > * sudo rule: add, remove option > * dnsrecord: add, update, delete > > https://fedorahosted.org/freeipa/ticket/2977 ACK. Some minor issues below: 1. The notification for operations involving multiple entries are a bit inconsistent. In the search facet deleting entries will generate "Selected entries were deleted." Enabling/disabling entries will generate " items were enabled/disabled". In the association facet, HBAC/sudo rules, adding/deleting entries will generate "Items added/removed." For sudo options it's "Option/Option(s) added/removed." 2. The notification for operations involving a single entry are a bit consistent too, but I think this is an existing issue. If you add an entry the notification will say " successfully added". It doesn't show the primary key which may be generated by the server. As a comparison if you delete an entry from the details page it will say "Deleted ''", and for update operation " updated." 3. Also existing issue, if you add using "Add and Add Another" the notification doesn't fade away, does it matter? Feel free to address this separately. -- Endi S. Dewata From rcritten at redhat.com Wed Sep 5 18:04:57 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 05 Sep 2012 14:04:57 -0400 Subject: [Freeipa-devel] [PATCH] 1041 pull in cachememsize logging In-Reply-To: <502A522A.1020507@redhat.com> References: <502A522A.1020507@redhat.com> Message-ID: <50479449.2060006@redhat.com> Rob Crittenden wrote: > 389-ds-base added logging if the entry cache is smaller than the > database so users will know they need to tune their DS install. Set this > as the minimum for IPA. > > rob Rebased patch. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-rcrit-1041-2-cachesize.patch Type: text/x-diff Size: 1609 bytes Desc: not available URL: From rcritten at redhat.com Wed Sep 5 18:06:09 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 05 Sep 2012 14:06:09 -0400 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <5037B79E.3070603@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> Message-ID: <50479491.1060200@redhat.com> Rob Crittenden wrote: > Martin Kosek wrote: >> On 07/05/2012 08:39 PM, Rob Crittenden wrote: >>> Martin Kosek wrote: >>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: >>>>> Deleting a replica can leave a replication vector (RUV) on the >>>>> other servers. >>>>> This can confuse things if the replica is re-added, and it also >>>>> causes the >>>>> server to calculate changes against a server that may no longer exist. >>>>> >>>>> 389-ds-base provides a new task that self-propogates itself to all >>>>> available >>>>> replicas to clean this RUV data. >>>>> >>>>> This patch will create this task at deletion time to hopefully >>>>> clean things up. >>>>> >>>>> It isn't perfect. If any replica is down or unavailable at the time >>>>> the >>>>> cleanruv task fires, and then comes back up, the old RUV data may be >>>>> re-propogated around. >>>>> >>>>> To make things easier in this case I've added two new commands to >>>>> ipa-replica-manage. The first lists the replication ids of all the >>>>> servers we >>>>> have a RUV for. Using this you can call clean_ruv with the >>>>> replication id of a >>>>> server that no longer exists to try the cleanallruv step again. >>>>> >>>>> This is quite dangerous though. If you run cleanruv against a >>>>> replica id that >>>>> does exist it can cause a loss of data. I believe I've put in >>>>> enough scary >>>>> warnings about this. >>>>> >>>>> rob >>>>> >>>> >>>> Good work there, this should make cleaning RUVs much easier than >>>> with the >>>> previous version. >>>> >>>> This is what I found during review: >>>> >>>> 1) list_ruv and clean_ruv command help in man is quite lost. I think >>>> it would >>>> help if we for example have all info for commands indented. This way >>>> user could >>>> simply over-look the new commands in the man page. >>>> >>>> >>>> 2) I would rename new commands to clean-ruv and list-ruv to make them >>>> consistent with the rest of the commands (re-initialize, force-sync). >>>> >>>> >>>> 3) It would be nice to be able to run clean_ruv command in an >>>> unattended way >>>> (for better testing), i.e. respect --force option as we already do for >>>> ipa-replica-manage del. This fix would aid test automation in the >>>> future. >>>> >>>> >>>> 4) (minor) The new question (and the del too) does not react too >>>> well for >>>> CTRL+D: >>>> >>>> # ipa-replica-manage clean_ruv 3 --force >>>> Clean the Replication Update Vector for >>>> vm-055.idm.lab.bos.redhat.com:389 >>>> >>>> Cleaning the wrong replica ID will cause that server to no >>>> longer replicate so it may miss updates while the process >>>> is running. It would need to be re-initialized to maintain >>>> consistency. Be very careful. >>>> Continue to clean? [no]: unexpected error: >>>> >>>> >>>> 5) Help for clean_ruv command without a required parameter is quite >>>> confusing >>>> as it reports that command is wrong and not the parameter: >>>> >>>> # ipa-replica-manage clean_ruv >>>> Usage: ipa-replica-manage [options] >>>> >>>> ipa-replica-manage: error: must provide a command [clean_ruv | >>>> force-sync | >>>> disconnect | connect | del | re-initialize | list | list_ruv] >>>> >>>> It seems you just forgot to specify the error message in the command >>>> definition >>>> >>>> >>>> 6) When the remote replica is down, the clean_ruv command fails with an >>>> unexpected error: >>>> >>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 >>>> Clean the Replication Update Vector for >>>> vm-055.idm.lab.bos.redhat.com:389 >>>> >>>> Cleaning the wrong replica ID will cause that server to no >>>> longer replicate so it may miss updates while the process >>>> is running. It would need to be re-initialized to maintain >>>> consistency. Be very careful. >>>> Continue to clean? [no]: y >>>> unexpected error: {'desc': 'Operations error'} >>>> >>>> >>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: >>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>> cleanAllRUV_task: failed >>>> to connect to repl agreement connection >>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, >>>> >>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>> tree,cn=config), error 105 >>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>> cleanAllRUV_task: replica >>>> (cn=meTovm-055.idm.lab. >>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>> >>>> >>>> tree, cn=config) has not been cleaned. You will need to rerun the >>>> CLEANALLRUV task on this replica. >>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>> cleanAllRUV_task: Task >>>> failed (1) >>>> >>>> In this case I think we should inform user that the command failed, >>>> possibly >>>> because of disconnected replicas and that they could enable the >>>> replicas and >>>> try again. >>>> >>>> >>>> 7) (minor) "pass" is now redundant in replication.py: >>>> + except ldap.INSUFFICIENT_ACCESS: >>>> + # We can't make the server we're removing read-only but >>>> + # this isn't a show-stopper >>>> + root_logger.debug("No permission to switch replica to >>>> read-only, >>>> continuing anyway") >>>> + pass >>>> >>> >>> I think this addresses everything. >>> >>> rob >> >> Thanks, almost there! I just found one more issue which needs to be fixed >> before we push: >> >> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >> Directory Manager password: >> >> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing >> removal >> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': "Can't >> contact LDAP server"} >> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >> >> There were issues removing a connection: %d format: a number is >> required, not str >> >> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': "Can't >> contact LDAP server"} >> >> This is a traceback I retrieved: >> Traceback (most recent call last): >> File "/sbin/ipa-replica-manage", line 425, in del_master >> del_link(realm, r, hostname, options.dirman_passwd, force=True) >> File "/sbin/ipa-replica-manage", line 271, in del_link >> repl1.cleanallruv(replica_id) >> File >> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >> line 1094, in cleanallruv >> root_logger.debug("Creating CLEANALLRUV task for replica id %d" % >> replicaId) >> >> >> The problem here is that you don't convert replica_id to int in this >> part: >> + replica_id = None >> + if repl2: >> + replica_id = repl2._get_replica_id(repl2.conn, None) >> + else: >> + servers = get_ruv(realm, replica1, dirman_passwd) >> + for (netloc, rid) in servers: >> + if netloc.startswith(replica2): >> + replica_id = rid >> + break >> >> Martin >> > > Updated patch using new mechanism in 389-ds-base. This should more > thoroughly clean out RUV data when a replica is being deleted, and > provide for a way to delete RUV data afterwards too if necessary. > > rob Rebased patch rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-rcrit-1031-4-cleanruv.patch Type: text/x-diff Size: 14038 bytes Desc: not available URL: From rcritten at redhat.com Wed Sep 5 18:08:15 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 05 Sep 2012 14:08:15 -0400 Subject: [Freeipa-devel] [PATCH] 1053 support 389-ds posix-winsync plugin Message-ID: <5047950F.6080709@redhat.com> Add support for the 389-ds posix winsync plugin. This plugin will sync the POSIX attributes from AD. We need to avoid trying to re-add them in our plugin. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-rcrit-1053-winsync.patch Type: text/x-diff Size: 6229 bytes Desc: not available URL: From rmeggins at redhat.com Wed Sep 5 18:13:47 2012 From: rmeggins at redhat.com (Rich Megginson) Date: Wed, 05 Sep 2012 12:13:47 -0600 Subject: [Freeipa-devel] [PATCH] 1053 support 389-ds posix-winsync plugin In-Reply-To: <5047950F.6080709@redhat.com> References: <5047950F.6080709@redhat.com> Message-ID: <5047965B.2080908@redhat.com> On 09/05/2012 12:08 PM, Rob Crittenden wrote: > Add support for the 389-ds posix winsync plugin. This plugin will sync > the POSIX attributes from AD. We need to avoid trying to re-add them > in our plugin. ack > > rob > > > _______________________________________________ > Freeipa-devel mailing list > Freeipa-devel at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From edewata at redhat.com Wed Sep 5 19:30:49 2012 From: edewata at redhat.com (Endi Sukma Dewata) Date: Wed, 05 Sep 2012 14:30:49 -0500 Subject: [Freeipa-devel] [PATCH] 204 Update of confirmation of actions In-Reply-To: <5044D541.2000904@redhat.com> References: <5044D541.2000904@redhat.com> Message-ID: <5047A869.4040906@redhat.com> On 9/3/2012 11:05 AM, Petr Vobornik wrote: > This patch is changing confirmation of actions according to ticket > #3035, see the ticket description. > > It does following changes: > * Confirmation of update action was removed. > * Action lists resets to first action (which is usually a NOP: '-- > select action --') on change of displayed entry. > * New confirmation dialog was implemented. It is used for action > confirmation. It is used in IPA.action to replace the call of > window.confirm(message). The old call is a modal window which blocks all > JS functionality and has different style than other dialogs in Web UI. > The new one has same design and doesn't block background operations. > > https://fedorahosted.org/freeipa/ticket/3035 Some issues: 1. None of the confirmation dialogs have a default button. If you hit Enter it will do nothing (except #2 below). All buttons are greyed out until you hover with mouse or focus with Tab. Is this intentional? I think usually a confirmation dialog would have a default button. 2. In the Users search facet the confirmation dialog doesn't show default button either, but if you hit Enter it will execute the operation. This is inconsistent with #1. -- Endi S. Dewata From rcritten at redhat.com Wed Sep 5 20:04:14 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 05 Sep 2012 16:04:14 -0400 Subject: [Freeipa-devel] [PATCH] 298 Add safe updates for objectClasses In-Reply-To: <504749C4.1090108@redhat.com> References: <503E0DD9.2020905@redhat.com> <503F6249.3060209@redhat.com> <5045EED8.70005@redhat.com> <50467742.5020900@redhat.com> <5046AF40.40408@redhat.com> <5046FDD3.9060200@redhat.com> <504749C4.1090108@redhat.com> Message-ID: <5047B03E.1090102@redhat.com> Martin Kosek wrote: > On 09/05/2012 09:22 AM, Martin Kosek wrote: >> On 09/05/2012 03:47 AM, Rob Crittenden wrote: >>> Rob Crittenden wrote: >>>> Martin Kosek wrote: >>>>> On 08/30/2012 02:53 PM, Rob Crittenden wrote: >>>>>> Martin Kosek wrote: >>>>>>> Current objectclass updates in a form of "replace" update instruction >>>>>>> dependent on exact match of the old object class specification in the >>>>>>> update instruction and the real value in LDAP. However, this >>>>>>> approach is >>>>>>> very error prone as object class definition can easily differ as for >>>>>>> example because of unexpected X-ORIGIN value. Such objectclass update >>>>>>> failures may lead to serious malfunctions later. >>>>>>> >>>>>>> Add new update instruction type "replaceoc" with the following format: >>>>>>> replaceoc:OID:new >>>>>>> This update instruction will always replace an objectclass with >>>>>>> specified OID with the new definition. >>>>>>> >>>>>>> https://fedorahosted.org/freeipa/ticket/2440 >>>>>> >>>>>> This works ok. Martin and I had a conversation in IRC about it. >>>>>> >>>>>> This moves from replacing a specific bit of schema with a new one, in >>>>>> all >>>>>> cases. I wonder if we should be more conservative and know what we're >>>>>> replacing >>>>>> in advance. >>>>>> >>>>>> rob >>>>>> >>>>> >>>>> You are right, I was too harsh when replacing the objectclasses. This >>>>> would >>>>> cause issues when LDAP update would be run on a replica with lower >>>>> version and >>>>> older objectclass definitions. >>>>> >>>>> I came up with an alternative solution and instead of always replacing >>>>> the >>>>> objectclass I rather reverted to old-OC:new-OC style which should be >>>>> safer. >>>>> Now, the LDAP updater always normalizes an objectclass before >>>>> comparing it >>>>> using python-ldap objectclass model. With this approach, objectclasses >>>>> differing only in X-ORIGIN or white spaces should match and be updated. >>>>> >>>>> Martin >>>>> >>>> >>>> NACK >>>> >>>> I think this: >>>> >>>> + for value in replaced_values: >>>> + entry_values.remove(old) >>>> >>>> Should be: >>>> >>>> + entry_values.remove(value) >> >> Right. Thanks for the fix. It only worked in my testing because I had no >> objectclass update which would differ in X-ORIGIN or whitespaces or case. I >> tried to mangle an update file and with this fix it did the update even if >> X-ORIGIN and whitespaces differed. >> >>>> >>>> I'm still doing other testing but this is what I've found so far. >>>> >>>> rob >>> >>> I did some more testing and it looks like this will do the trick. >>> >>> I also found a place where the schema was left as unicode and causing it to >>> blow up inside python-ldap. Here is the diff on my working instance: >>> >>> diff -u ipaserver/install/ldapupdate.py >>> /usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py >>> --- ipaserver/install/ldapupdate.py 2012-09-04 16:59:33.210688723 -0400 >>> +++ /usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py 2012-09-04 >>> 21:47:01.583574375 -0400 >>> @@ -643,7 +643,7 @@ >>> self.debug('replace: no match for replaced >>> ObjectClass "%s"', old) >>> continue >>> for value in replaced_values: >>> - entry_values.remove(old) >>> + entry_values.remove(value) >>> else: >>> entry_values.remove(old) >>> entry_values.append(new) >>> @@ -772,7 +772,11 @@ >>> updated = False >>> changes = self.conn.generateModList(entry.origDataDict(), >>> entry.toDict()) >>> if (entry.dn == DN(('cn', 'schema'))): >>> - updated = self.is_schema_updated(entry.toDict()) >>> + d = dict() >>> + e = entry.toDict() >>> + for k,v in e.items(): >>> + d[k] = [str(x) for x in v] >>> + updated = self.is_schema_updated(d) >>> else: >>> if len(changes) >= 1: >>> updated = True >>> >>> rob >>> >> >> I did not hit this error during testing, but at least it won't harm. Sending an >> updated patch. >> > > Sending a slightly updated patch which now makes sure that we always pass > objectclass of str (and not unicode) type to ObjectClass model. This should > make it more bullet-proof. > > Martin > ACK, pushed to master and ipa-3-0 rob From rcritten at redhat.com Wed Sep 5 20:20:08 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 05 Sep 2012 16:20:08 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <5044A40B.6070604@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502B86F5.4070008@redhat.com> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> Message-ID: <5047B3F8.4060406@redhat.com> Martin Kosek wrote: > On 08/31/2012 04:53 PM, Petr Viktorin wrote: >> On 08/28/2012 03:40 PM, Petr Viktorin wrote: >>> On 08/17/2012 06:04 PM, Ade Lee wrote: >>>> On Fri, 2012-08-17 at 09:34 -0400, Ade Lee wrote: >>>>> On Thu, 2012-08-16 at 18:45 +0200, Martin Kosek wrote: >>>>>> On 08/16/2012 01:28 PM, Ade Lee wrote: >>>>>>> Patch attached this time. I should know better than to do this in the >>>>>>> middle of the night .. >>>>>>> >>>>>>> On Thu, 2012-08-16 at 09:12 +0200, Martin Kosek wrote: >>>>>>>> On 08/16/2012 07:53 AM, Ade Lee wrote: >>>>>>>>> On Wed, 2012-08-15 at 23:41 -0400, Ade Lee wrote: >>>>>>>>>> On Wed, 2012-08-15 at 16:34 +0200, Martin Kosek wrote: >>>>>>>>>>> On 08/15/2012 03:54 PM, Ade Lee wrote: >>>>>>>>>>>> On Wed, 2012-08-15 at 13:24 +0200, Martin Kosek wrote: >>>>>>>>>>>>> On 08/08/2012 10:05 PM, Ade Lee wrote: >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Dogtag 10 is being released on f18, and has a number of >>>>>>>>>>>>>> changes that >>>>>>>>>>>>>> will affect IPA. In particular, the following changes will >>>>>>>>>>>>>> affect >>>>>>>>>>>>>> current IPA code. >>>>>>>>>>>>>> >>>>>>>>>>>>>> * The directory layout of the dogtag instance has changed. >>>>>>>>>>>>>> Instead of >>>>>>>>>>>>>> using separate tomcat instances to host different >>>>>>>>>>>>>> subsystems, the >>>>>>>>>>>>>> standard dogtag installation will allow one to install a CA. >>>>>>>>>>>>>> KRA, OCSP >>>>>>>>>>>>>> and TKS within the same instance. There have been >>>>>>>>>>>>>> corresponding changes >>>>>>>>>>>>>> in the directory layout, as well as the default instance name >>>>>>>>>>>>>> (pki-tomcat instead of pki-ca), and startup daemon >>>>>>>>>>>>>> (pki-tomcatd, instead >>>>>>>>>>>>>> of pki-cad, pki-krad etc.) >>>>>>>>>>>>>> >>>>>>>>>>>>>> * The default instance will use only four ports (HTTPS, >>>>>>>>>>>>>> HTTP, AJP and >>>>>>>>>>>>>> tomcat shutdown port) rather than the 6 previously used. >>>>>>>>>>>>>> The default >>>>>>>>>>>>>> ports will be changed to the standard tomcat ports. As >>>>>>>>>>>>>> these ports are >>>>>>>>>>>>>> local to the ipa server machine, this should not cause too much >>>>>>>>>>>>>> disruption. >>>>>>>>>>>>>> >>>>>>>>>>>>>> * There is a new single step installer written in python. >>>>>>>>>>>>>> (pkispawn/destroy) vs. pkicreate/pkisilent/pkiremove. >>>>>>>>>>>>>> >>>>>>>>>>>>>> * Dogtag 10 runs on tomcat7 - with a new corresponding >>>>>>>>>>>>>> version of >>>>>>>>>>>>>> tomcatjss. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The attached patch integrates all the above changes in IPA >>>>>>>>>>>>>> installation >>>>>>>>>>>>>> and maintenance code. Once the patch is applied, users will >>>>>>>>>>>>>> be able to: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. run ipa-server-install to completion on f18 with dogtag 10. >>>>>>>>>>>>>> 2. install a new replica on f18 on dogtag 10. >>>>>>>>>>>>>> 3. upgrade an f17 machine with an existing IPA instance to >>>>>>>>>>>>>> f18/ dogtag >>>>>>>>>>>>>> 10 - and have that old-style dogtag instance continue to run >>>>>>>>>>>>>> correctly. >>>>>>>>>>>>>> This will require the installation of the latest version of >>>>>>>>>>>>>> tomcatjss as >>>>>>>>>>>>>> well as the installation of tomcat6. The old-style instance >>>>>>>>>>>>>> will >>>>>>>>>>>>>> continue to use tomcat6. >>>>>>>>>>>>>> 4. in addition, the new cert renewal code has been patched >>>>>>>>>>>>>> and should >>>>>>>>>>>>>> continue to work. >>>>>>>>>>>>>> >>>>>>>>>>>>>> What is not yet completed / supported: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. Installation with an external CA is not yet completed in >>>>>>>>>>>>>> the new >>>>>>>>>>>>>> installer. We plan to complete this soon. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2. There is some IPA upgrade code that has not yet been touched >>>>>>>>>>>>>> (install/tools/ipa-upgradeconfig). >>>>>>>>>>>>>> >>>>>>>>>>>>>> 3. A script needs to be written to allow admins to convert >>>>>>>>>>>>>> their >>>>>>>>>>>>>> old-style dogtag instances to new style instances, as well >>>>>>>>>>>>>> as code to >>>>>>>>>>>>>> periodically prompt admins to do this. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 4. Installation of old-style instances using >>>>>>>>>>>>>> pkicreate/pkisilent on >>>>>>>>>>>>>> dogtag 10 will no longer be supported, and will be disabled >>>>>>>>>>>>>> soon. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 5. The pki-selinux policy has been updated to reflect these >>>>>>>>>>>>>> changes, >>>>>>>>>>>>>> but is still in flux. In fact, it is our intention to place >>>>>>>>>>>>>> the dogtag >>>>>>>>>>>>>> selinux policy in the base selinux policy for f18. In the >>>>>>>>>>>>>> meantime, it >>>>>>>>>>>>>> may be necessary to run installs in permissive mode. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The dogtag 10 code will be released shortly into f18. Prior >>>>>>>>>>>>>> to that >>>>>>>>>>>>>> though, we have placed the new dogtag 10 and tomcatjss code >>>>>>>>>>>>>> in a >>>>>>>>>>>>>> developer repo that is located at >>>>>>>>>>>>>> http://nkinder.fedorapeople.org/dogtag-devel/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> Testing can be done on both f18 and f17 - although the >>>>>>>>>>>>>> target platform - >>>>>>>>>>>>>> and the only platform for which official builds will be >>>>>>>>>>>>>> created is f18. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Ade >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Ade, >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks for the patch, I started with review and integration >>>>>>>>>>>>> tests (currently >>>>>>>>>>>>> running on Fedora 17 with Nathan's repo). >>>>>>>>>>>>> >>>>>>>>>>>>> Installation on single master was smooth, it worked just >>>>>>>>>>>>> fine, even with >>>>>>>>>>>>> enforced SELinux, without any error - kudos to you and the >>>>>>>>>>>>> whole dogtag team. >>>>>>>>>>>>> The resulting logs and the structure of your log directory >>>>>>>>>>>>> seems improved. I >>>>>>>>>>>>> believe that the brand new Python installers will make it >>>>>>>>>>>>> easier to debug >>>>>>>>>>>>> issues with dogtag installation when they come. When I tried >>>>>>>>>>>>> our unit tests or >>>>>>>>>>>>> some simple cert operation, it worked fine as well. >>>>>>>>>>>>> >>>>>>>>>>>>> Now the bad news, or rather few issues and suggestions I found: >>>>>>>>>>>>> >>>>>>>>>>>>> 1) As we already discussed on IRC, tomcat 7 was not pulled >>>>>>>>>>>>> automatically on >>>>>>>>>>>>> Fedora 17 when I updated pki-ca, you somewhere miss a Requires. >>>>>>>>>>>>> >>>>>>>>>>>> We have a dogtag patch that is currently in review that will >>>>>>>>>>>> address >>>>>>>>>>>> this. Once this is in, tomcatjss >=7.0.0 will be required for >>>>>>>>>>>> f17+, >>>>>>>>>>>> rather than f18+ >>>>>>>>>>>> >>>>>>>>>>>>> 2) I had installed IPA with dogtag10 on master. However, CA >>>>>>>>>>>>> installation on a >>>>>>>>>>>>> replica (ipa-ca-install) with dogtag9 failed - is this >>>>>>>>>>>>> expectable? >>>>>>>>>>>>> >>>>>>>>>>>> Yes. The current IPA patch is designed to work with dogtag 10 >>>>>>>>>>>> only, >>>>>>>>>>>> which will be officially available on f18+. So if you update >>>>>>>>>>>> to dogtag >>>>>>>>>>>> 10, you must have this patch and visa versa. We probably need >>>>>>>>>>>> to add >>>>>>>>>>>> the relevant requires to IPA to enforce this. >>>>>>>>>>>> >>>>>>>>>>>> If you have an existing dogtag 9 instance, and you upgrade to >>>>>>>>>>>> the new >>>>>>>>>>>> dogtag 10 and patched IPA, then that instance will continue to >>>>>>>>>>>> work. >>>>>>>>>>>> But any new instances would be created using dogtag 10. >>>>>>>>>>>> >>>>>>>>>>>>> 3) I had installed IPA with dogtag10 on master. Replica had >>>>>>>>>>>>> dogtag10 as well >>>>>>>>>>>>> and I got the following error: >>>>>>>>>>>>> >>>>>>>>>>>>> # ipa-ca-install >>>>>>>>>>>>> /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg >>>>>>>>>>>>> ... >>>>>>>>>>>>> Configuring certificate server: Estimated time 3 minutes 30 >>>>>>>>>>>>> seconds >>>>>>>>>>>>> [1/14]: creating certificate server user >>>>>>>>>>>>> [2/14]: configuring certificate server instance >>>>>>>>>>>>> >>>>>>>>>>>>> Your system may be partly configured. >>>>>>>>>>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up. >>>>>>>>>>>>> >>>>>>>>>>>>> Unexpected error - see /var/log/ipareplica-ca-install.log for >>>>>>>>>>>>> details: >>>>>>>>>>>>> IOError: [Errno 2] No such file or directory: >>>>>>>>>>>>> '/var/lib/pki/pki-tomcat/alias/ca_backup_keys.p12' >>>>>>>>>>>>> >>>>>>>>>>>>> Root cause: >>>>>>>>>>>>> ... >>>>>>>>>>>>> File >>>>>>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", >>>>>>>>>>>>> line >>>>>>>>>>>>> 625, in __spawn_instance >>>>>>>>>>>>> "/root/cacert.p12") >>>>>>>>>>>>> ... >>>>>>>>>>>>> >>>>>>>>>>>> I need to look into this. I had fixed ipa-replica-install, >>>>>>>>>>>> rather than >>>>>>>>>>>> ipa-ca-install to create replicas. I didn't know ipa-ca-install >>>>>>>>>>>> existed! Should not be too bad to fix though - most likely >>>>>>>>>>>> just need to >>>>>>>>>>>> move files to the right place. >>>>>>>>>>> >>>>>>>>>>> Ok, thanks! Btw. CA on replica can be either installed during >>>>>>>>>>> ipa-replica-install time (when --setup-ca option is passed, you >>>>>>>>>>> probably used >>>>>>>>>>> that one) and the aforementioned ipa-ca-install run after >>>>>>>>>>> ipa-replica-install. >>>>>>>>>>> >>>>>>>>>> I will be testing this out again. As ipa-ca-install uses the >>>>>>>>>> same code >>>>>>>>>> as ipa-replica-install, I would have expected it to work. Did >>>>>>>>>> you try >>>>>>>>>> it with selinux in permissive mode? >>>>>>>>>> >>>>>>>>> OK -- I tested this out and realized that between the time I >>>>>>>>> initially >>>>>>>>> tested this and your tests, we had changed a URL >>>>>>>>> from /ca/pki/installer/installToken to >>>>>>>>> /ca/rest/installer/installToken, >>>>>>>>> but I forgot to update ipa-pki-proxy.conf. >>>>>>>>> >>>>>>>>> The new patch has been rebased and changes the relevant patch. With >>>>>>>>> this, the CA replica installs correctly. >>>>>>>> >>>>>>>> Umh, where can I find the new rebased patch? :-) >>>>>>>> >>>>>>>>> >>>>>>>>> There was one issue that I encountered. I have seen this once >>>>>>>>> before >>>>>>>>> and will need to investigate further. IPA sometimes restarts the >>>>>>>>> dogtag >>>>>>>>> instance by doing systemctl stop pki-tomcatd.target. and then >>>>>>>>> systemctl >>>>>>>>> start pki-tomcatd.target. I have noticed that occasionally the >>>>>>>>> start >>>>>>>>> invocation fails, while the stop and restart invocations succeed >>>>>>>>> just >>>>>>>>> fine. This makes absolutely no sense because restart is essentially >>>>>>>>> just stop and then start. >>>>>>>>> >>>>>>>>> I think this is actually a bug in systemd. If I reboot the >>>>>>>>> machine, it >>>>>>>>> all works perfectly. If we can't figure out whats going on with >>>>>>>>> systemd, we can always replace this start/stop with >>>>>>>>> starting/stopping >>>>>>>>> the service directly. (pki-tomcatd at pki-tomcat.service). That >>>>>>>>> seems to >>>>>>>>> work all the time with no problems. >>>>>>>>> >>>>>>>>> I suggest we leave this fix (if needed) for a separate patch. >>>>>>>> >>>>>>>> Ok, I will see if I hit this issue again. Btw. is it recommended >>>>>>>> in systemd to >>>>>>>> use target units this way? I thought that only service files are >>>>>>>> meant to be >>>>>>>> used for manipulation with a service. As you said yourself, using >>>>>>>> service unit >>>>>>>> file for restart worked every time. >>>>>>>> >>>>>>>> Martin >>>>>>> >>>>>> >>>>>> Now, I tested the rebased patch with VMs with permissive SELinux. >>>>>> IPA server >>>>>> install was OK, as always, but replica installation ended up with >>>>>> error. >>>>>> Although it got much further than before: >>>>>> >>>>>> # ipa-ca-install >>>>>> /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg >>>>>> ... >>>>>> [3/3]: restarting directory server >>>>>> done configuring pkids. >>>>>> Configuring certificate server: Estimated time 3 minutes 30 seconds >>>>>> [1/14]: creating certificate server user >>>>>> [2/14]: configuring certificate server instance >>>>>> [3/14]: disabling nonces >>>>>> [4/14]: importing CA chain to RA certificate database >>>>>> [5/14]: fixing RA database permissions >>>>>> [6/14]: setting up signing cert profile >>>>>> [7/14]: set up CRL publishing >>>>>> [8/14]: set certificate subject base >>>>>> [9/14]: enabling Subject Key Identifier >>>>>> [10/14]: configuring certificate server to start on boot >>>>>> [11/14]: configure certmonger for renewals >>>>>> [12/14]: configure clone certificate renewals >>>>>> [13/14]: configure Server-Cert certificate renewal >>>>>> [14/14]: Configure HTTP to proxy connections >>>>>> done configuring pki-tomcatd. >>>>>> Restarting the directory and certificate servers >>>>>> >>>>>> Your system may be partly configured. >>>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up. >>>>>> >>>>>> It seems like that CA on a replica did not start and so a check for >>>>>> port 8080 >>>>>> failed. Attached logs (pki-ca-logs.tgz) may provide more information >>>>>> to this issue. >>>>>> >>>>> This is the systemd issue I mentioned before. I will submit a patch >>>>> which will change the restart mechanism to restart the service and not >>>>> the target. >>>>>> >>>> >>>> Patch attached. This patch should be applied on top of the large patch >>>> (being reviewed). >>>> >>>>>> Now I am also testing upgrade from IPA+dogtag9 to PatchedIPA+dogtag10 >>>>>> (permissive SELinux). Now, the service was upgraded smoothly, CA >>>>>> service >>>>>> could be restarted without failure. However, certificate operations now >>>>>> did not work: >>>>>> >>>>>> # ipactl restart >>>>>> Restarting Directory Service >>>>>> Restarting KDC Service >>>>>> Restarting KPASSWD Service >>>>>> Restarting MEMCACHE Service >>>>>> Restarting HTTP Service >>>>>> Restarting CA Service >>>>>> # ipactl status >>>>>> Directory Service: RUNNING >>>>>> KDC Service: RUNNING >>>>>> KPASSWD Service: RUNNING >>>>>> MEMCACHE Service: RUNNING >>>>>> HTTP Service: RUNNING >>>>>> CA Service: RUNNING >>>>>> # ipa cert-show 1 >>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to >>>>>> communicate with CMS (Internal Server Error) >>>>>> >>>>>> Attaching logs for this case as well (ipa-ca-logs-upgrade.tgz). >>>>>> >>>>> The reason for this is that the old dogtag instances are missing: >>>>> 1. a new parameter in CS.cfg >>>>> 2. a couple of symlinks >>>>> >>>>> I plan to fix this in the dogtag code. Specifically, >>>>> 1. I will modify the dogtag code to provide a default value in the case >>>>> where the parameter does not exist. >>>>> 2. I plan to add a function to the dogtag startup scripts to check and >>>>> remake any required symlinks. >>>>> >>>>> Both of these changes are in the dogtag code, and do not affect this >>>>> patch. As a workaround, to confirm that the upgrade actually works, do >>>>> the following manual steps on your upgraded instance: >>>>> >>>>> 1. Add the following parameter to CS.cfg >>>>> pidDir=/var/run >>>>> >>>>> 2. Add the following links; >>>>> cd /var/lib/pki-ca/common/lib >>>>> ln -s /usr/share/java/apache-commons-codec.jar >>>>> apache-commons-codec.jar >>>>> ln -s /usr/share/java/log4j.jar log4j.jar >>>>> >>>>> 3 Restart the dogtag instance >>>>> >>>>>> HTH, >>>>>> Martin >>>>> >>>>> >>> >>> The attached patch conditionalizes the changes, so that IPA supports >>> both Dogtag versions. >>> >>> I didn't put it in platform code for two reasons >>> - One platform (f17) can have either Dogtag version installed >>> - I didn't want to conflict with Timo Aaltonen's "platformizing" work. >>> According to his WIP patches, he plans to move the whole dogtag module >>> into platform code. >>> >>> I used the existence of /usr/bin/pkispawn as a test for Dogtag 10. If >>> you know a better way, please comment. >>> >>> The dogtag_version option is now always added to >>> >>> I've reverted the changes to install/ui/test/data/ipa_init.json, so >>> you'll have to change the ports manually when testing the UI with Dogtag >>> 10. >>> >>> >>> >> >> Attaching a patch with fixed ipa-pki-proxy.conf, as discussed on IRC. >> > > This approach looks good. I did not hit any regression on F17 with dogtag9, or > clean installs of IPA+dogtag10. I think we are getting close to pushing this code. > > The only issues I hit so far are for upgraded dogtag9 instances (on F17): > > 1) The service did not start for me. There were some SELinux AVCs and even when > I disabled SELinux the instance still did not start (logs attached). > > 2) Uninstall was also not clean, we leave some dogtag installation states for > upgraded dogtag9 instance: > > # ipa-server-install --uninstall --unattended > Shutting down all IPA services > Removing IPA client configuration > Unconfiguring ntpd > Unconfiguring CA directory server > Unconfiguring web server > Unconfiguring krb5kdc > Unconfiguring kadmin > Unconfiguring directory server > Unconfiguring ipa_memcached > ipa : ERROR Some installation state for pki-cad has not been > restored, see /var/lib/ipa/sysrestore/sysrestore.state > > /var/lib/ipa/sysrestore/sysrestore.state: > [pki-cad] > enabled = False Ade is working on a new build to address the dogtag upgrade issues. The left over state should be removed when we drop the separate instance in the dogtag 9 -> 10 upgrade. I'm not completely sure reading the patch who actually does this part, is this handled by dogtag? rob From nalin at redhat.com Wed Sep 5 20:43:59 2012 From: nalin at redhat.com (Nalin Dahyabhai) Date: Wed, 5 Sep 2012 16:43:59 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <1346244513.2539.172.camel@aleeredhat.laptop> References: <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <502E6BFD.90400@redhat.com> <503B6A98.2040902@redhat.com> <503B6BF8.8090909@redhat.com> <503B78E5.10008@redhat.com> <1346212418.2539.169.camel@aleeredhat.laptop> <1346244513.2539.172.camel@aleeredhat.laptop> Message-ID: <20120905204359.GC22642@redhat.com> On Wed, Aug 29, 2012 at 08:48:32AM -0400, Ade Lee wrote: > Incidentally, I ran this in permmissive selinux mode. The following > rules are required to be added: > > #============= certmonger_t ============== > corenet_tcp_connect_http_cache_port(certmonger_t) > files_read_var_lib_symlinks(certmonger_t) On my system, "semanage port -l" shows me: http_cache_port_t tcp 8080, 8118, 10001-10010 Are these ports already labeled this way for Dogtag, or is it a coincidental overlap with some other package? If it's an overlap, it might be better to switch to using ports which aren't already labeled for use in policy that applies to some other package. If not, please open a bug against the selinux-policy component to get these accesses added to the set that's allowed by the default policy. Nalin From rcritten at redhat.com Wed Sep 5 20:57:34 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 05 Sep 2012 16:57:34 -0400 Subject: [Freeipa-devel] [PATCH] 83 Use OpenSSH-style public keys as the preferred format of SSH public keys In-Reply-To: <50463229.5030305@redhat.com> References: <50463229.5030305@redhat.com> Message-ID: <5047BCBE.1090304@redhat.com> Jan Cholasta wrote: > Hi, > > this patch changes the format of the sshpubkey parameter to the format > used by OpenSSH (see sshd(8)). > > Public keys in the old format (raw RFC 4253 blob) are automatically > converted to OpenSSH-style public keys. OpenSSH-style public keys are > now stored in LDAP. > > Changed sshpubkeyfp to be an output parameter, as that is what it > actually is. > > Allow parameter normalizers to be used on values of any type, not just > unicode, so that public key blobs (which are str) can be normalized to > OpenSSH-style public keys. > > Note that you need a SSSD build including > > (SSSD 1.9.0beta7-to-be) in order to make OpenSSH integration actually > work with OpenSSH-style public keys. > > > > > Honza NACK. I think a bunch of tests are needed for this. Because you abstracted out the pubkey class it should be straightforward to add a bunch of class-based unit tests on it. There are also no user or host-based tests, either for adding or managing keys. I tested backwards compatibility with 2.2 and the initial tests are mixed. I installed 2.2 and created a 3.0 clone from it, including your patch. I added a user in 3.0 with a key and it added ok, but on the 2.2 side it returns the entire base64 encoded blob of key type, key and comment, which I presume is unusable. At least things don't blow up. The reverse works fine. An old-style key added to 2.2 appears to work fine in 3.0, we just lack a comment. On the 2.2 server: $ ipa user-show tuser1 --all | grep -i ssh Base-64 encoded SSH public key: c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDNUQyRTI2dHU5YXM2cHhlUVlSdUgzelYyUDUzMjFpR1U5aC9XNElpd0tGSGlOc2p5cXFyemhCUFB3am83dGlYRDlHbUo1M25KS21OTGd0K01XUnFTZEx2R0V3NjM3SkVTWEpGL0VWeUxvZEFWRGltdXFRVkNLWjBRcm1kYjErRUg1VGRrd3ByOExyd0g1a0RzMEVpcGc2c0xoRUZ5NzMvaXNjRkJqcmk0NGxSU1BZNXFHTWFLOVE0cjY1WFEyaytlZ1RDQnBNZnc0b0J6Mzh0ZHVEVVE2bW9XNFhQSnhZeWJ3MGFDMnRUK2RBOU42WndFSFZXREUzdzg0bHRHa0JRZFRaKzViRnBFdlladm9PbkZXdDlNZFIzYVd6UklnY1o5VDlySDFFT2Z3eE5zWVRCLzRjTmg3dS9adGxnMVV0Z1VteWN3TkpMTUYrMTNzNTl2OFFpSFogcmNyaXRAZWRzZWwuZ3JleW9hay5jb20= $ python Python 2.7.3 (default, Jul 24 2012, 10:05:38) [GCC 4.7.0 20120507 (Red Hat 4.7.0-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import base64 >>> s = 'c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDNUQyRTI2dHU5YXM2cHhlUVlSdUgzelYyUDUzMjFpR1U5aC9XNElpd0tGSGlOc2p5cXFyemhCUFB3am83dGlYRDlHbUo1M25KS21OTGd0K01XUnFTZEx2R0V3NjM3SkVTWEpGL0VWeUxvZEFWRGltdXFRVkNLWjBRcm1kYjErRUg1VGRrd3ByOExyd0g1a0RzMEVpcGc2c0xoRUZ5NzMvaXNjRkJqcmk0NGxSU1BZNXFHTWFLOVE0cjY1WFEyaytlZ1RDQnBNZnc0b0J6Mzh0ZHVEVVE2bW9XNFhQSnhZeWJ3MGFDMnRUK2RBOU42WndFSFZXREUzdzg0bHRHa0JRZFRaKzViRnBFdlladm9PbkZXdDlNZFIzYVd6UklnY1o5VDlySDFFT2Z3eE5zWVRCLzRjTmg3dS9adGxnMVV0Z1VteWN3TkpMTUYrMTNzNTl2OFFpSFogcmNyaXRAZWRzZWwuZ3JleW9hay5jb20=' >>> base64.b64decode(s) 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5D2E26tu9as6pxeQYRuH3zV2P5321iGU9h/W4IiwKFHiNsjyqqrzhBPPwjo7tiXD9GmJ53nJKmNLgt+MWRqSdLvGEw637JESXJF/EVyLodAVDimuqQVCKZ0Qrmdb1+EH5Tdkwpr8LrwH5kDs0Eipg6sLhEFy73/iscFBjri44lRSPY5qGMaK9Q4r65XQ2k+egTCBpMfw4oBz38tduDUQ6moW4XPJxYybw0aC2tT+dA9N6ZwEHVWDE3w84ltGkBQdTZ+5bFpEvYZvoOnFWt9MdR3aWzRIgcZ9T9rH1EOfwxNsYTB/4cNh7u/Ztlg1UtgUmycwNJLMF+13s59v8QiHZ rcrit at edsel.greyoak.com' Now show an old style key: $ ipa user-show tuser2 --all | grep -i ssh Base-64 encoded SSH public key: AAAAB3NzaC1yc2EAAAADAQABAAABAQCbRLyizFGyfucNRnHpWdUG8dBD7W2PfvTQ42k+LmAdUFudTytO89oTRXcVEYMDL42OyRth12JRMUjYTEmFwo9a9Mb7cP8+bo7N2lV4iCB0CUybcZARF0MV6NeYhhWlC9DV40nkqs3Goe8X8tMPXn/HZn8Rz33703w8K/G6STnN0txhAT4tY7D3e0DA9UY87wNnpJ7dXoJqMXRv2dRgmUnGih/8cLHypyxBoLoL8qR9cWxAf/Cs+qQmsk15lzIGQUAJwwXBBjbnXKwykEeHjTHsvjd7zzC1cWtz5Zz/8aop7AsVwaBqb9u+5dVOMxdzLGD24NKTjhtG86ADU4Mpnlb5 rob From alee at redhat.com Wed Sep 5 21:08:12 2012 From: alee at redhat.com (Ade Lee) Date: Wed, 05 Sep 2012 17:08:12 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <20120905204359.GC22642@redhat.com> References: <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <502E6BFD.90400@redhat.com> <503B6A98.2040902@redhat.com> <503B6BF8.8090909@redhat.com> <503B78E5.10008@redhat.com> <1346212418.2539.169.camel@aleeredhat.laptop> <1346244513.2539.172.camel@aleeredhat.laptop> <20120905204359.GC22642@redhat.com> Message-ID: <1346879293.24731.39.camel@aleeredhat.laptop> On Wed, 2012-09-05 at 16:43 -0400, Nalin Dahyabhai wrote: > On Wed, Aug 29, 2012 at 08:48:32AM -0400, Ade Lee wrote: > > Incidentally, I ran this in permmissive selinux mode. The following > > rules are required to be added: > > > > #============= certmonger_t ============== > > corenet_tcp_connect_http_cache_port(certmonger_t) > > files_read_var_lib_symlinks(certmonger_t) > > On my system, "semanage port -l" shows me: > http_cache_port_t tcp 8080, 8118, 10001-10010 > > Are these ports already labeled this way for Dogtag, or is it a > coincidental overlap with some other package? If it's an overlap, > it might be better to switch to using ports which aren't already labeled > for use in policy that applies to some other package. > We have specifically chosen to use what would be the default ports for tomcat. These ports are already labeled as you have described above. We have adjusted our selinux policy to handle that. In fact, we are now extending a tomcat selinux domain provided by the system policies, and this tomcat domain allows access to those ports. > If not, please open a bug against the selinux-policy component to get > these accesses added to the set that's allowed by the default policy. > I can open a bug. > Nalin From edewata at redhat.com Wed Sep 5 21:35:53 2012 From: edewata at redhat.com (Endi Sukma Dewata) Date: Wed, 05 Sep 2012 16:35:53 -0500 Subject: [Freeipa-devel] [PATCH] 205 Reflect API change of SSH store in Web UI In-Reply-To: <504709CF.4010201@redhat.com> References: <504709CF.4010201@redhat.com> Message-ID: <5047C5B9.3080809@redhat.com> On 9/5/2012 3:14 AM, Petr Vobornik wrote: > Format of ipasshpubkey in users and hosts changed from BYTES to STR. Web > UI no longer gets the value as base64 encoded string in an object. > > Label was changed to reflect that the key don't have to be plain base64 > encoded blob. > > https://fedorahosted.org/freeipa/ticket/2989 > > Note: freeipa-jcholast-83-openssh-style-public-keys.patch should be applied ACK. Possible improvements: 1. Right now if you click 'Add' SSH public key you'd have to click 'Show/Set key' to enter the value. We probably could make it such that when you click 'Add' it will open the input dialog immediately. This way we can avoid an incomplete state where a slot for a new key is added but it's empty. 2. If we do #1 the 'New: key set/not set' label can be changed to 'New key'. The 'Modified' can be changed to 'Modified key'. 3. The 'Show/Set key' probably can be changed to 'View/Edit' to be more consistent with host/service certificate. -- Endi S. Dewata From edewata at redhat.com Wed Sep 5 21:40:12 2012 From: edewata at redhat.com (Endi Sukma Dewata) Date: Wed, 05 Sep 2012 16:40:12 -0500 Subject: [Freeipa-devel] [PATCH] 206-208 Fixed number parameters MIN boundary validation in Web UI In-Reply-To: <50475CC4.5010205@redhat.com> References: <50475CC4.5010205@redhat.com> Message-ID: <5047C6BC.6000005@redhat.com> On 9/5/2012 9:08 AM, Petr Vobornik wrote: > Integers were missing most of minimum checks and Decimals boundaries > weren't checked at all in Web UI. > > First part is done in ipalib, second in Web UI. > > 1) [PATCH] 206 Fixed metadata serialization of Numbers and DNs: > There were following problems: > 1. DNs and Decimals weren't properly serialized. Serialization output > was object with empty __base64__ attribute. It was fixed by converting > them to string. > 2. numberical values equal to 0 were excluded from metadata. It broke > many of minvalue checks in Web UI. Now excluding only None and False > values as initially intended. ACK. > 2)[PATCH] 207 Added decimal checks to metadata validator: > Metadata validator didn't have check for decimal values. It was added. ACK. > 3)[PATCH] 208 Generated metadata for testing updated > Testing metadata needs to be updated because of fix in json serialization. ACK. -- Endi S. Dewata From nalin at redhat.com Wed Sep 5 21:42:34 2012 From: nalin at redhat.com (Nalin Dahyabhai) Date: Wed, 5 Sep 2012 17:42:34 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <1346879293.24731.39.camel@aleeredhat.laptop> References: <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <502E6BFD.90400@redhat.com> <503B6A98.2040902@redhat.com> <503B6BF8.8090909@redhat.com> <503B78E5.10008@redhat.com> <1346212418.2539.169.camel@aleeredhat.laptop> <1346244513.2539.172.camel@aleeredhat.laptop> <20120905204359.GC22642@redhat.com> <1346879293.24731.39.camel@aleeredhat.laptop> Message-ID: <20120905214234.GD22642@redhat.com> On Wed, Sep 05, 2012 at 05:08:12PM -0400, Ade Lee wrote: > On Wed, 2012-09-05 at 16:43 -0400, Nalin Dahyabhai wrote: > > On Wed, Aug 29, 2012 at 08:48:32AM -0400, Ade Lee wrote: > > > Incidentally, I ran this in permmissive selinux mode. The following > > > rules are required to be added: > > > > > > #============= certmonger_t ============== > > > corenet_tcp_connect_http_cache_port(certmonger_t) > > > files_read_var_lib_symlinks(certmonger_t) > > > > On my system, "semanage port -l" shows me: > > http_cache_port_t tcp 8080, 8118, 10001-10010 > > > > Are these ports already labeled this way for Dogtag, or is it a > > coincidental overlap with some other package? If it's an overlap, > > it might be better to switch to using ports which aren't already labeled > > for use in policy that applies to some other package. > > We have specifically chosen to use what would be the default ports for > tomcat. These ports are already labeled as you have described above. > We have adjusted our selinux policy to handle that. In fact, we are now > extending a tomcat selinux domain provided by the system policies, and > this tomcat domain allows access to those ports. My thinking, based on the name, is that the policy expects this set of ports to be used by squid, and actual HTTP caches, rather than arbitrary servlet containers. But then I suppose the policy maintainer will know better. Please CC me on the policy bug so that I can keep an eye on it. Thanks, Nalin From simo at redhat.com Wed Sep 5 21:44:12 2012 From: simo at redhat.com (Simo Sorce) Date: Wed, 05 Sep 2012 17:44:12 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <1346879293.24731.39.camel@aleeredhat.laptop> References: <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <502E6BFD.90400@redhat.com> <503B6A98.2040902@redhat.com> <503B6BF8.8090909@redhat.com> <503B78E5.10008@redhat.com> <1346212418.2539.169.camel@aleeredhat.laptop> <1346244513.2539.172.camel@aleeredhat.laptop> <20120905204359.GC22642@redhat.com> <1346879293.24731.39.camel@aleeredhat.laptop> Message-ID: <1346881452.1150.537.camel@willson.li.ssimo.org> On Wed, 2012-09-05 at 17:08 -0400, Ade Lee wrote: > On Wed, 2012-09-05 at 16:43 -0400, Nalin Dahyabhai wrote: > > On Wed, Aug 29, 2012 at 08:48:32AM -0400, Ade Lee wrote: > > > Incidentally, I ran this in permmissive selinux mode. The following > > > rules are required to be added: > > > > > > #============= certmonger_t ============== > > > corenet_tcp_connect_http_cache_port(certmonger_t) > > > files_read_var_lib_symlinks(certmonger_t) > > > > On my system, "semanage port -l" shows me: > > http_cache_port_t tcp 8080, 8118, 10001-10010 > > > > Are these ports already labeled this way for Dogtag, or is it a > > coincidental overlap with some other package? If it's an overlap, > > it might be better to switch to using ports which aren't already labeled > > for use in policy that applies to some other package. > > > We have specifically chosen to use what would be the default ports for > tomcat. These ports are already labeled as you have described above. > We have adjusted our selinux policy to handle that. In fact, we are now > extending a tomcat selinux domain provided by the system policies, and > this tomcat domain allows access to those ports. > > > If not, please open a bug against the selinux-policy component to get > > these accesses added to the set that's allowed by the default policy. > > > I can open a bug. Ade, how will the selinux policy be handled in an upgrade scenario ? If I understand correctly you are dropping custom selinux policies from dogtag 10 and relying on system policy going forward, so what will keep the right labels for the old ports in an upgrade scenario ? Or will the rpm upgrade also change ports ? Is this properly handled on the ipa part yet ? Simo. -- Simo Sorce * Red Hat, Inc * New York From edewata at redhat.com Wed Sep 5 22:46:57 2012 From: edewata at redhat.com (Endi Sukma Dewata) Date: Wed, 05 Sep 2012 17:46:57 -0500 Subject: [Freeipa-devel] [PATCH] 209 Fixed problem while deleting entry with unsaved changes In-Reply-To: <5047691D.6010108@redhat.com> References: <5047691D.6010108@redhat.com> Message-ID: <5047D661.6030700@redhat.com> On 9/5/2012 10:00 AM, Petr Vobornik wrote: > While deleting an entry it now resets a facet if there are unsaved > changes. It prevents pop up of various error dialogs when UI tries to > redirect to search page after successful delete. > > https://fedorahosted.org/freeipa/ticket/3047 ACK. -- Endi S. Dewata From alee at redhat.com Thu Sep 6 00:48:37 2012 From: alee at redhat.com (Ade Lee) Date: Wed, 05 Sep 2012 20:48:37 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <1346881452.1150.537.camel@willson.li.ssimo.org> References: <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <502E6BFD.90400@redhat.com> <503B6A98.2040902@redhat.com> <503B6BF8.8090909@redhat.com> <503B78E5.10008@redhat.com> <1346212418.2539.169.camel@aleeredhat.laptop> <1346244513.2539.172.camel@aleeredhat.laptop> <20120905204359.GC22642@redhat.com> <1346879293.24731.39.camel@aleeredhat.laptop> <1346881452.1150.537.camel@willson.li.ssimo.org> Message-ID: <1346892518.24731.49.camel@aleeredhat.laptop> On Wed, 2012-09-05 at 17:44 -0400, Simo Sorce wrote: > On Wed, 2012-09-05 at 17:08 -0400, Ade Lee wrote: > > On Wed, 2012-09-05 at 16:43 -0400, Nalin Dahyabhai wrote: > > > On Wed, Aug 29, 2012 at 08:48:32AM -0400, Ade Lee wrote: > > > > Incidentally, I ran this in permmissive selinux mode. The following > > > > rules are required to be added: > > > > > > > > #============= certmonger_t ============== > > > > corenet_tcp_connect_http_cache_port(certmonger_t) > > > > files_read_var_lib_symlinks(certmonger_t) > > > > > > On my system, "semanage port -l" shows me: > > > http_cache_port_t tcp 8080, 8118, 10001-10010 > > > > > > Are these ports already labeled this way for Dogtag, or is it a > > > coincidental overlap with some other package? If it's an overlap, > > > it might be better to switch to using ports which aren't already labeled > > > for use in policy that applies to some other package. > > > > > We have specifically chosen to use what would be the default ports for > > tomcat. These ports are already labeled as you have described above. > > We have adjusted our selinux policy to handle that. In fact, we are now > > extending a tomcat selinux domain provided by the system policies, and > > this tomcat domain allows access to those ports. > > > > > If not, please open a bug against the selinux-policy component to get > > > these accesses added to the set that's allowed by the default policy. > > > > > I can open a bug. > > Ade, how will the selinux policy be handled in an upgrade scenario ? > If I understand correctly you are dropping custom selinux policies from > dogtag 10 and relying on system policy going forward, so what will keep > the right labels for the old ports in an upgrade scenario ? > Or will the rpm upgrade also change ports ? Is this properly handled on > the ipa part yet ? > > Simo. > To be clear, there will still be a selinux policy for dogtag, but it will be delivered as part of the system policies. The system selinux policy will contain alias definitions for all the old types. So for example, pki_ca_var_lib_t etc. will be aliased to the new type pki_tomcat_var_lib_t. The old ports will continue to be defined in the system policy but they will also be aliased to the ports in the new policy. Its not all defined yet - which is why we still need selinux permissive -- but we're working to get the system policy done soon. Ade From alee at redhat.com Thu Sep 6 01:01:16 2012 From: alee at redhat.com (Ade Lee) Date: Wed, 05 Sep 2012 21:01:16 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <5047B3F8.4060406@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502B86F5.4070008@redhat.com> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> Message-ID: <1346893277.24731.60.camel@aleeredhat.laptop> On Wed, 2012-09-05 at 16:20 -0400, Rob Crittenden wrote: > Martin Kosek wrote: > > On 08/31/2012 04:53 PM, Petr Viktorin wrote: > >> On 08/28/2012 03:40 PM, Petr Viktorin wrote: > >>> On 08/17/2012 06:04 PM, Ade Lee wrote: > >>>> On Fri, 2012-08-17 at 09:34 -0400, Ade Lee wrote: > >>>>> On Thu, 2012-08-16 at 18:45 +0200, Martin Kosek wrote: > >>>>>> On 08/16/2012 01:28 PM, Ade Lee wrote: > >>>>>>> Patch attached this time. I should know better than to do this in the > >>>>>>> middle of the night .. > >>>>>>> > >>>>>>> On Thu, 2012-08-16 at 09:12 +0200, Martin Kosek wrote: > >>>>>>>> On 08/16/2012 07:53 AM, Ade Lee wrote: > >>>>>>>>> On Wed, 2012-08-15 at 23:41 -0400, Ade Lee wrote: > >>>>>>>>>> On Wed, 2012-08-15 at 16:34 +0200, Martin Kosek wrote: > >>>>>>>>>>> On 08/15/2012 03:54 PM, Ade Lee wrote: > >>>>>>>>>>>> On Wed, 2012-08-15 at 13:24 +0200, Martin Kosek wrote: > >>>>>>>>>>>>> On 08/08/2012 10:05 PM, Ade Lee wrote: > >>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Dogtag 10 is being released on f18, and has a number of > >>>>>>>>>>>>>> changes that > >>>>>>>>>>>>>> will affect IPA. In particular, the following changes will > >>>>>>>>>>>>>> affect > >>>>>>>>>>>>>> current IPA code. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> * The directory layout of the dogtag instance has changed. > >>>>>>>>>>>>>> Instead of > >>>>>>>>>>>>>> using separate tomcat instances to host different > >>>>>>>>>>>>>> subsystems, the > >>>>>>>>>>>>>> standard dogtag installation will allow one to install a CA. > >>>>>>>>>>>>>> KRA, OCSP > >>>>>>>>>>>>>> and TKS within the same instance. There have been > >>>>>>>>>>>>>> corresponding changes > >>>>>>>>>>>>>> in the directory layout, as well as the default instance name > >>>>>>>>>>>>>> (pki-tomcat instead of pki-ca), and startup daemon > >>>>>>>>>>>>>> (pki-tomcatd, instead > >>>>>>>>>>>>>> of pki-cad, pki-krad etc.) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> * The default instance will use only four ports (HTTPS, > >>>>>>>>>>>>>> HTTP, AJP and > >>>>>>>>>>>>>> tomcat shutdown port) rather than the 6 previously used. > >>>>>>>>>>>>>> The default > >>>>>>>>>>>>>> ports will be changed to the standard tomcat ports. As > >>>>>>>>>>>>>> these ports are > >>>>>>>>>>>>>> local to the ipa server machine, this should not cause too much > >>>>>>>>>>>>>> disruption. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> * There is a new single step installer written in python. > >>>>>>>>>>>>>> (pkispawn/destroy) vs. pkicreate/pkisilent/pkiremove. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> * Dogtag 10 runs on tomcat7 - with a new corresponding > >>>>>>>>>>>>>> version of > >>>>>>>>>>>>>> tomcatjss. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The attached patch integrates all the above changes in IPA > >>>>>>>>>>>>>> installation > >>>>>>>>>>>>>> and maintenance code. Once the patch is applied, users will > >>>>>>>>>>>>>> be able to: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 1. run ipa-server-install to completion on f18 with dogtag 10. > >>>>>>>>>>>>>> 2. install a new replica on f18 on dogtag 10. > >>>>>>>>>>>>>> 3. upgrade an f17 machine with an existing IPA instance to > >>>>>>>>>>>>>> f18/ dogtag > >>>>>>>>>>>>>> 10 - and have that old-style dogtag instance continue to run > >>>>>>>>>>>>>> correctly. > >>>>>>>>>>>>>> This will require the installation of the latest version of > >>>>>>>>>>>>>> tomcatjss as > >>>>>>>>>>>>>> well as the installation of tomcat6. The old-style instance > >>>>>>>>>>>>>> will > >>>>>>>>>>>>>> continue to use tomcat6. > >>>>>>>>>>>>>> 4. in addition, the new cert renewal code has been patched > >>>>>>>>>>>>>> and should > >>>>>>>>>>>>>> continue to work. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> What is not yet completed / supported: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 1. Installation with an external CA is not yet completed in > >>>>>>>>>>>>>> the new > >>>>>>>>>>>>>> installer. We plan to complete this soon. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 2. There is some IPA upgrade code that has not yet been touched > >>>>>>>>>>>>>> (install/tools/ipa-upgradeconfig). > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 3. A script needs to be written to allow admins to convert > >>>>>>>>>>>>>> their > >>>>>>>>>>>>>> old-style dogtag instances to new style instances, as well > >>>>>>>>>>>>>> as code to > >>>>>>>>>>>>>> periodically prompt admins to do this. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 4. Installation of old-style instances using > >>>>>>>>>>>>>> pkicreate/pkisilent on > >>>>>>>>>>>>>> dogtag 10 will no longer be supported, and will be disabled > >>>>>>>>>>>>>> soon. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 5. The pki-selinux policy has been updated to reflect these > >>>>>>>>>>>>>> changes, > >>>>>>>>>>>>>> but is still in flux. In fact, it is our intention to place > >>>>>>>>>>>>>> the dogtag > >>>>>>>>>>>>>> selinux policy in the base selinux policy for f18. In the > >>>>>>>>>>>>>> meantime, it > >>>>>>>>>>>>>> may be necessary to run installs in permissive mode. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The dogtag 10 code will be released shortly into f18. Prior > >>>>>>>>>>>>>> to that > >>>>>>>>>>>>>> though, we have placed the new dogtag 10 and tomcatjss code > >>>>>>>>>>>>>> in a > >>>>>>>>>>>>>> developer repo that is located at > >>>>>>>>>>>>>> http://nkinder.fedorapeople.org/dogtag-devel/ > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Testing can be done on both f18 and f17 - although the > >>>>>>>>>>>>>> target platform - > >>>>>>>>>>>>>> and the only platform for which official builds will be > >>>>>>>>>>>>>> created is f18. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>> Ade > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hi Ade, > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks for the patch, I started with review and integration > >>>>>>>>>>>>> tests (currently > >>>>>>>>>>>>> running on Fedora 17 with Nathan's repo). > >>>>>>>>>>>>> > >>>>>>>>>>>>> Installation on single master was smooth, it worked just > >>>>>>>>>>>>> fine, even with > >>>>>>>>>>>>> enforced SELinux, without any error - kudos to you and the > >>>>>>>>>>>>> whole dogtag team. > >>>>>>>>>>>>> The resulting logs and the structure of your log directory > >>>>>>>>>>>>> seems improved. I > >>>>>>>>>>>>> believe that the brand new Python installers will make it > >>>>>>>>>>>>> easier to debug > >>>>>>>>>>>>> issues with dogtag installation when they come. When I tried > >>>>>>>>>>>>> our unit tests or > >>>>>>>>>>>>> some simple cert operation, it worked fine as well. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Now the bad news, or rather few issues and suggestions I found: > >>>>>>>>>>>>> > >>>>>>>>>>>>> 1) As we already discussed on IRC, tomcat 7 was not pulled > >>>>>>>>>>>>> automatically on > >>>>>>>>>>>>> Fedora 17 when I updated pki-ca, you somewhere miss a Requires. > >>>>>>>>>>>>> > >>>>>>>>>>>> We have a dogtag patch that is currently in review that will > >>>>>>>>>>>> address > >>>>>>>>>>>> this. Once this is in, tomcatjss >=7.0.0 will be required for > >>>>>>>>>>>> f17+, > >>>>>>>>>>>> rather than f18+ > >>>>>>>>>>>> > >>>>>>>>>>>>> 2) I had installed IPA with dogtag10 on master. However, CA > >>>>>>>>>>>>> installation on a > >>>>>>>>>>>>> replica (ipa-ca-install) with dogtag9 failed - is this > >>>>>>>>>>>>> expectable? > >>>>>>>>>>>>> > >>>>>>>>>>>> Yes. The current IPA patch is designed to work with dogtag 10 > >>>>>>>>>>>> only, > >>>>>>>>>>>> which will be officially available on f18+. So if you update > >>>>>>>>>>>> to dogtag > >>>>>>>>>>>> 10, you must have this patch and visa versa. We probably need > >>>>>>>>>>>> to add > >>>>>>>>>>>> the relevant requires to IPA to enforce this. > >>>>>>>>>>>> > >>>>>>>>>>>> If you have an existing dogtag 9 instance, and you upgrade to > >>>>>>>>>>>> the new > >>>>>>>>>>>> dogtag 10 and patched IPA, then that instance will continue to > >>>>>>>>>>>> work. > >>>>>>>>>>>> But any new instances would be created using dogtag 10. > >>>>>>>>>>>> > >>>>>>>>>>>>> 3) I had installed IPA with dogtag10 on master. Replica had > >>>>>>>>>>>>> dogtag10 as well > >>>>>>>>>>>>> and I got the following error: > >>>>>>>>>>>>> > >>>>>>>>>>>>> # ipa-ca-install > >>>>>>>>>>>>> /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg > >>>>>>>>>>>>> ... > >>>>>>>>>>>>> Configuring certificate server: Estimated time 3 minutes 30 > >>>>>>>>>>>>> seconds > >>>>>>>>>>>>> [1/14]: creating certificate server user > >>>>>>>>>>>>> [2/14]: configuring certificate server instance > >>>>>>>>>>>>> > >>>>>>>>>>>>> Your system may be partly configured. > >>>>>>>>>>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Unexpected error - see /var/log/ipareplica-ca-install.log for > >>>>>>>>>>>>> details: > >>>>>>>>>>>>> IOError: [Errno 2] No such file or directory: > >>>>>>>>>>>>> '/var/lib/pki/pki-tomcat/alias/ca_backup_keys.p12' > >>>>>>>>>>>>> > >>>>>>>>>>>>> Root cause: > >>>>>>>>>>>>> ... > >>>>>>>>>>>>> File > >>>>>>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", > >>>>>>>>>>>>> line > >>>>>>>>>>>>> 625, in __spawn_instance > >>>>>>>>>>>>> "/root/cacert.p12") > >>>>>>>>>>>>> ... > >>>>>>>>>>>>> > >>>>>>>>>>>> I need to look into this. I had fixed ipa-replica-install, > >>>>>>>>>>>> rather than > >>>>>>>>>>>> ipa-ca-install to create replicas. I didn't know ipa-ca-install > >>>>>>>>>>>> existed! Should not be too bad to fix though - most likely > >>>>>>>>>>>> just need to > >>>>>>>>>>>> move files to the right place. > >>>>>>>>>>> > >>>>>>>>>>> Ok, thanks! Btw. CA on replica can be either installed during > >>>>>>>>>>> ipa-replica-install time (when --setup-ca option is passed, you > >>>>>>>>>>> probably used > >>>>>>>>>>> that one) and the aforementioned ipa-ca-install run after > >>>>>>>>>>> ipa-replica-install. > >>>>>>>>>>> > >>>>>>>>>> I will be testing this out again. As ipa-ca-install uses the > >>>>>>>>>> same code > >>>>>>>>>> as ipa-replica-install, I would have expected it to work. Did > >>>>>>>>>> you try > >>>>>>>>>> it with selinux in permissive mode? > >>>>>>>>>> > >>>>>>>>> OK -- I tested this out and realized that between the time I > >>>>>>>>> initially > >>>>>>>>> tested this and your tests, we had changed a URL > >>>>>>>>> from /ca/pki/installer/installToken to > >>>>>>>>> /ca/rest/installer/installToken, > >>>>>>>>> but I forgot to update ipa-pki-proxy.conf. > >>>>>>>>> > >>>>>>>>> The new patch has been rebased and changes the relevant patch. With > >>>>>>>>> this, the CA replica installs correctly. > >>>>>>>> > >>>>>>>> Umh, where can I find the new rebased patch? :-) > >>>>>>>> > >>>>>>>>> > >>>>>>>>> There was one issue that I encountered. I have seen this once > >>>>>>>>> before > >>>>>>>>> and will need to investigate further. IPA sometimes restarts the > >>>>>>>>> dogtag > >>>>>>>>> instance by doing systemctl stop pki-tomcatd.target. and then > >>>>>>>>> systemctl > >>>>>>>>> start pki-tomcatd.target. I have noticed that occasionally the > >>>>>>>>> start > >>>>>>>>> invocation fails, while the stop and restart invocations succeed > >>>>>>>>> just > >>>>>>>>> fine. This makes absolutely no sense because restart is essentially > >>>>>>>>> just stop and then start. > >>>>>>>>> > >>>>>>>>> I think this is actually a bug in systemd. If I reboot the > >>>>>>>>> machine, it > >>>>>>>>> all works perfectly. If we can't figure out whats going on with > >>>>>>>>> systemd, we can always replace this start/stop with > >>>>>>>>> starting/stopping > >>>>>>>>> the service directly. (pki-tomcatd at pki-tomcat.service). That > >>>>>>>>> seems to > >>>>>>>>> work all the time with no problems. > >>>>>>>>> > >>>>>>>>> I suggest we leave this fix (if needed) for a separate patch. > >>>>>>>> > >>>>>>>> Ok, I will see if I hit this issue again. Btw. is it recommended > >>>>>>>> in systemd to > >>>>>>>> use target units this way? I thought that only service files are > >>>>>>>> meant to be > >>>>>>>> used for manipulation with a service. As you said yourself, using > >>>>>>>> service unit > >>>>>>>> file for restart worked every time. > >>>>>>>> > >>>>>>>> Martin > >>>>>>> > >>>>>> > >>>>>> Now, I tested the rebased patch with VMs with permissive SELinux. > >>>>>> IPA server > >>>>>> install was OK, as always, but replica installation ended up with > >>>>>> error. > >>>>>> Although it got much further than before: > >>>>>> > >>>>>> # ipa-ca-install > >>>>>> /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg > >>>>>> ... > >>>>>> [3/3]: restarting directory server > >>>>>> done configuring pkids. > >>>>>> Configuring certificate server: Estimated time 3 minutes 30 seconds > >>>>>> [1/14]: creating certificate server user > >>>>>> [2/14]: configuring certificate server instance > >>>>>> [3/14]: disabling nonces > >>>>>> [4/14]: importing CA chain to RA certificate database > >>>>>> [5/14]: fixing RA database permissions > >>>>>> [6/14]: setting up signing cert profile > >>>>>> [7/14]: set up CRL publishing > >>>>>> [8/14]: set certificate subject base > >>>>>> [9/14]: enabling Subject Key Identifier > >>>>>> [10/14]: configuring certificate server to start on boot > >>>>>> [11/14]: configure certmonger for renewals > >>>>>> [12/14]: configure clone certificate renewals > >>>>>> [13/14]: configure Server-Cert certificate renewal > >>>>>> [14/14]: Configure HTTP to proxy connections > >>>>>> done configuring pki-tomcatd. > >>>>>> Restarting the directory and certificate servers > >>>>>> > >>>>>> Your system may be partly configured. > >>>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up. > >>>>>> > >>>>>> It seems like that CA on a replica did not start and so a check for > >>>>>> port 8080 > >>>>>> failed. Attached logs (pki-ca-logs.tgz) may provide more information > >>>>>> to this issue. > >>>>>> > >>>>> This is the systemd issue I mentioned before. I will submit a patch > >>>>> which will change the restart mechanism to restart the service and not > >>>>> the target. > >>>>>> > >>>> > >>>> Patch attached. This patch should be applied on top of the large patch > >>>> (being reviewed). > >>>> > >>>>>> Now I am also testing upgrade from IPA+dogtag9 to PatchedIPA+dogtag10 > >>>>>> (permissive SELinux). Now, the service was upgraded smoothly, CA > >>>>>> service > >>>>>> could be restarted without failure. However, certificate operations now > >>>>>> did not work: > >>>>>> > >>>>>> # ipactl restart > >>>>>> Restarting Directory Service > >>>>>> Restarting KDC Service > >>>>>> Restarting KPASSWD Service > >>>>>> Restarting MEMCACHE Service > >>>>>> Restarting HTTP Service > >>>>>> Restarting CA Service > >>>>>> # ipactl status > >>>>>> Directory Service: RUNNING > >>>>>> KDC Service: RUNNING > >>>>>> KPASSWD Service: RUNNING > >>>>>> MEMCACHE Service: RUNNING > >>>>>> HTTP Service: RUNNING > >>>>>> CA Service: RUNNING > >>>>>> # ipa cert-show 1 > >>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to > >>>>>> communicate with CMS (Internal Server Error) > >>>>>> > >>>>>> Attaching logs for this case as well (ipa-ca-logs-upgrade.tgz). > >>>>>> > >>>>> The reason for this is that the old dogtag instances are missing: > >>>>> 1. a new parameter in CS.cfg > >>>>> 2. a couple of symlinks > >>>>> > >>>>> I plan to fix this in the dogtag code. Specifically, > >>>>> 1. I will modify the dogtag code to provide a default value in the case > >>>>> where the parameter does not exist. > >>>>> 2. I plan to add a function to the dogtag startup scripts to check and > >>>>> remake any required symlinks. > >>>>> > >>>>> Both of these changes are in the dogtag code, and do not affect this > >>>>> patch. As a workaround, to confirm that the upgrade actually works, do > >>>>> the following manual steps on your upgraded instance: > >>>>> > >>>>> 1. Add the following parameter to CS.cfg > >>>>> pidDir=/var/run > >>>>> > >>>>> 2. Add the following links; > >>>>> cd /var/lib/pki-ca/common/lib > >>>>> ln -s /usr/share/java/apache-commons-codec.jar > >>>>> apache-commons-codec.jar > >>>>> ln -s /usr/share/java/log4j.jar log4j.jar > >>>>> > >>>>> 3 Restart the dogtag instance > >>>>> > >>>>>> HTH, > >>>>>> Martin > >>>>> > >>>>> > >>> > >>> The attached patch conditionalizes the changes, so that IPA supports > >>> both Dogtag versions. > >>> > >>> I didn't put it in platform code for two reasons > >>> - One platform (f17) can have either Dogtag version installed > >>> - I didn't want to conflict with Timo Aaltonen's "platformizing" work. > >>> According to his WIP patches, he plans to move the whole dogtag module > >>> into platform code. > >>> > >>> I used the existence of /usr/bin/pkispawn as a test for Dogtag 10. If > >>> you know a better way, please comment. > >>> > >>> The dogtag_version option is now always added to > >>> > >>> I've reverted the changes to install/ui/test/data/ipa_init.json, so > >>> you'll have to change the ports manually when testing the UI with Dogtag > >>> 10. > >>> > >>> > >>> > >> > >> Attaching a patch with fixed ipa-pki-proxy.conf, as discussed on IRC. > >> > > > > This approach looks good. I did not hit any regression on F17 with dogtag9, or > > clean installs of IPA+dogtag10. I think we are getting close to pushing this code. > > > > The only issues I hit so far are for upgraded dogtag9 instances (on F17): > > > > 1) The service did not start for me. There were some SELinux AVCs and even when > > I disabled SELinux the instance still did not start (logs attached). > > > > 2) Uninstall was also not clean, we leave some dogtag installation states for > > upgraded dogtag9 instance: > > > > # ipa-server-install --uninstall --unattended > > Shutting down all IPA services > > Removing IPA client configuration > > Unconfiguring ntpd > > Unconfiguring CA directory server > > Unconfiguring web server > > Unconfiguring krb5kdc > > Unconfiguring kadmin > > Unconfiguring directory server > > Unconfiguring ipa_memcached > > ipa : ERROR Some installation state for pki-cad has not been > > restored, see /var/lib/ipa/sysrestore/sysrestore.state > > > > /var/lib/ipa/sysrestore/sysrestore.state: > > [pki-cad] > > enabled = False > > Ade is working on a new build to address the dogtag upgrade issues. > > The left over state should be removed when we drop the separate instance > in the dogtag 9 -> 10 upgrade. I'm not completely sure reading the patch > who actually does this part, is this handled by dogtag? > > rob > The build is available on the developer repo. Just do a yum update. As for this leftover state, this is an ipa file and such is never handled by dogtag. It must be handled somewhere within the ipa code. Ade From mkosek at redhat.com Thu Sep 6 07:34:01 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 06 Sep 2012 09:34:01 +0200 Subject: [Freeipa-devel] [PATCHES] Remove dependencies to private samba libs In-Reply-To: <1644106786.2826837.1346673798761.JavaMail.root@redhat.com> References: <1644106786.2826837.1346673798761.JavaMail.root@redhat.com> Message-ID: <504851E9.2020408@redhat.com> On 09/03/2012 02:03 PM, Alexander Bokovoy wrote: > ----- Original Message ----- >> From: "Sumit Bose" >> To: "freeipa-devel" >> Sent: Sunday, September 2, 2012 8:04:22 PM >> Subject: [Freeipa-devel] [PATCHES] Remove dependencies to private samba libs >> >> Hi, >> >> private samba libraries will become a new version with every new >> samba >> release. To avoid rebuilding of IPA whenever a new samba version is >> released the following series of patches removes the dependencies to >> private samba libraries by replacing the related calls with others. >> >> This should fix https://fedorahosted.org/freeipa/ticket/3013 > ACK to all of them, I checked and they work correctly. > > I had concerns about strdup upper patch but talloc_strdup_upper() avoids reallocation of memory for our use case (storing security key). > Pushed them all to master, ipa-3-0. Martin From pvoborni at redhat.com Thu Sep 6 08:07:21 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Thu, 06 Sep 2012 10:07:21 +0200 Subject: [Freeipa-devel] [PATCH] 203 Notify success on add, delete and update In-Reply-To: <50478B1C.8010504@redhat.com> References: <5044B22D.7030602@redhat.com> <50478B1C.8010504@redhat.com> Message-ID: <504859B9.7020600@redhat.com> On 09/05/2012 07:25 PM, Endi Sukma Dewata wrote: > On 9/3/2012 8:35 AM, Petr Vobornik wrote: >> Notification of success was added to: >> * details facet: update >> * association facet and association widget: add, delete items >> * attribute facet: delete items (notification of add should be handled >> in entity adder dialog) >> * sudo rule: add, remove option >> * dnsrecord: add, update, delete >> >> https://fedorahosted.org/freeipa/ticket/2977 > > ACK. Some minor issues below: Pushed to master. > > 1. The notification for operations involving multiple entries are a bit > inconsistent. In the search facet deleting entries will generate > "Selected entries were deleted." Enabling/disabling entries will > generate " items were enabled/disabled". In the association facet, > HBAC/sudo rules, adding/deleting entries will generate "Items > added/removed." For sudo options it's "Option/Option(s) added/removed." > To change notification we should also review confirmation messages. For example delete message reflect 'Do you want to delete selected items'. I guess if we want to change it, we should make them more specific and avoid general words like items or entries. Btw which of the two is better? > 2. The notification for operations involving a single entry are a bit > consistent too, but I think this is an existing issue. If you add an > entry the notification will say " successfully added". It > doesn't show the primary key which may be generated by the server. As a > comparison if you delete an entry from the details page it will say > "Deleted ''", and for update operation " > updated." > In adder dialog we should probably used the message sent by the server, as we do in the other cases. > 3. Also existing issue, if you add using "Add and Add Another" the > notification doesn't fade away, does it matter? I don't think we should change it. Because in the time when it would fade away user will be filling some fields. User might be surprised by sudden move in the dialog. > > Feel free to address this separately. > -- Petr Vobornik From pvoborni at redhat.com Thu Sep 6 08:07:48 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Thu, 06 Sep 2012 10:07:48 +0200 Subject: [Freeipa-devel] [PATCH] 199 Permissions: select only applicable options on type change In-Reply-To: <50478AE6.9090202@redhat.com> References: <5034CDD9.2050900@redhat.com> <503CD9AA.8090207@redhat.com> <50448D95.2080605@redhat.com> <50478AE6.9090202@redhat.com> Message-ID: <504859D4.7080208@redhat.com> On 09/05/2012 07:24 PM, Endi Sukma Dewata wrote: > On 9/3/2012 5:59 AM, Petr Vobornik wrote: >> Updated patch attached. > > ACK. > Pushed to master and ipa-3-0. -- Petr Vobornik From pvoborni at redhat.com Thu Sep 6 08:08:01 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Thu, 06 Sep 2012 10:08:01 +0200 Subject: [Freeipa-devel] [PATCH] 197 Fixed search in HBAC test In-Reply-To: <50478ADF.7070100@redhat.com> References: <50339523.1070909@redhat.com> <503CD8EC.6050503@redhat.com> <50449453.2010404@redhat.com> <50478ADF.7070100@redhat.com> Message-ID: <504859E1.6040202@redhat.com> On 09/05/2012 07:24 PM, Endi Sukma Dewata wrote: > On 9/3/2012 6:28 AM, Petr Vobornik wrote: >>>> b) force refresh when searching with unchanged filter >> >> I did (b). Updated patch attached. >> >> I don't want to implement 'expiration date' at the moment. It's too >> widespread change. Maybe in FreeIPA 3.2. > > ACK. > Pushed to master and ipa-3-0. -- Petr Vobornik From pvoborni at redhat.com Thu Sep 6 08:32:01 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Thu, 06 Sep 2012 10:32:01 +0200 Subject: [Freeipa-devel] [PATCH] 209 Fixed problem while deleting entry with unsaved changes In-Reply-To: <5047D661.6030700@redhat.com> References: <5047691D.6010108@redhat.com> <5047D661.6030700@redhat.com> Message-ID: <50485F81.8070202@redhat.com> On 09/06/2012 12:46 AM, Endi Sukma Dewata wrote: > On 9/5/2012 10:00 AM, Petr Vobornik wrote: >> While deleting an entry it now resets a facet if there are unsaved >> changes. It prevents pop up of various error dialogs when UI tries to >> redirect to search page after successful delete. >> >> https://fedorahosted.org/freeipa/ticket/3047 > > ACK. > Pushed to master, ipa-3-0. -- Petr Vobornik From pvoborni at redhat.com Thu Sep 6 08:32:26 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Thu, 06 Sep 2012 10:32:26 +0200 Subject: [Freeipa-devel] [PATCH] 206-208 Fixed number parameters MIN boundary validation in Web UI In-Reply-To: <5047C6BC.6000005@redhat.com> References: <50475CC4.5010205@redhat.com> <5047C6BC.6000005@redhat.com> Message-ID: <50485F9A.3020106@redhat.com> On 09/05/2012 11:40 PM, Endi Sukma Dewata wrote: > On 9/5/2012 9:08 AM, Petr Vobornik wrote: >> Integers were missing most of minimum checks and Decimals boundaries >> weren't checked at all in Web UI. >> >> First part is done in ipalib, second in Web UI. >> >> 1) [PATCH] 206 Fixed metadata serialization of Numbers and DNs: >> There were following problems: >> 1. DNs and Decimals weren't properly serialized. Serialization output >> was object with empty __base64__ attribute. It was fixed by converting >> them to string. >> 2. numberical values equal to 0 were excluded from metadata. It broke >> many of minvalue checks in Web UI. Now excluding only None and False >> values as initially intended. > > ACK. > >> 2)[PATCH] 207 Added decimal checks to metadata validator: >> Metadata validator didn't have check for decimal values. It was added. > > ACK. > >> 3)[PATCH] 208 Generated metadata for testing updated >> Testing metadata needs to be updated because of fix in json >> serialization. > > ACK. > Pushed all 3 patches to master, ipa-3-0. -- Petr Vobornik From pvoborni at redhat.com Thu Sep 6 09:24:39 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Thu, 06 Sep 2012 11:24:39 +0200 Subject: [Freeipa-devel] [PATCH] 204 Update of confirmation of actions In-Reply-To: <5047A869.4040906@redhat.com> References: <5044D541.2000904@redhat.com> <5047A869.4040906@redhat.com> Message-ID: <50486BD7.2050200@redhat.com> On 09/05/2012 09:30 PM, Endi Sukma Dewata wrote: > On 9/3/2012 11:05 AM, Petr Vobornik wrote: >> This patch is changing confirmation of actions according to ticket >> #3035, see the ticket description. >> >> It does following changes: >> * Confirmation of update action was removed. >> * Action lists resets to first action (which is usually a NOP: '-- >> select action --') on change of displayed entry. >> * New confirmation dialog was implemented. It is used for action >> confirmation. It is used in IPA.action to replace the call of >> window.confirm(message). The old call is a modal window which blocks all >> JS functionality and has different style than other dialogs in Web UI. >> The new one has same design and doesn't block background operations. >> >> https://fedorahosted.org/freeipa/ticket/3035 > > Some issues: > > 1. None of the confirmation dialogs have a default button. If you hit > Enter it will do nothing (except #2 below). All buttons are greyed out > until you hover with mouse or focus with Tab. Is this intentional? I > think usually a confirmation dialog would have a default button. > > 2. In the Users search facet the confirmation dialog doesn't show > default button either, but if you hit Enter it will execute the > operation. This is inconsistent with #1. > Root of the problem is that old custom dialogs which serves for confirmation (batch delete, various certificate and unprovisioning dialogs) or other dialogs (password change, adder dialogs) don't have confirmation by enter. Only the new dialog has confirmation by Enter. At the moment this new dialog is used only for mass enable/disable. We might: a) remove confirmation confirmation by pressing Enter key from the new dialog to be consistent. IMO wrong. b) add this confirmation too all dialogs c) change old confirm dialogs to inherit from IPA.confirm_dialog to receive this functionality IMO, for most dialogs c) would be better but for some (highly customized) b). It should be probably done in other patch, maybe it the context of this ticket. There is also: https://fedorahosted.org/freeipa/ticket/2910 ([Web UI] Use "Enter" to confirm add dialogs) -- Petr Vobornik From mkosek at redhat.com Thu Sep 6 09:31:05 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 06 Sep 2012 11:31:05 +0200 Subject: [Freeipa-devel] [PATCH] 1041 pull in cachememsize logging In-Reply-To: <50479449.2060006@redhat.com> References: <502A522A.1020507@redhat.com> <50479449.2060006@redhat.com> Message-ID: <50486D59.509@redhat.com> On 09/05/2012 08:04 PM, Rob Crittenden wrote: > Rob Crittenden wrote: >> 389-ds-base added logging if the entry cache is smaller than the >> database so users will know they need to tune their DS install. Set this >> as the minimum for IPA. >> >> rob > > Rebased patch. > > rob > It looks like a changelog entry from your patch 1031-4 slipped in to this patch. Otherwise it is an obvious patch to ACK. Martin From mkosek at redhat.com Thu Sep 6 09:51:17 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 06 Sep 2012 11:51:17 +0200 Subject: [Freeipa-devel] [PATCH] 304 Allow localhost in zone ACIs Message-ID: <50487215.70600@redhat.com> Loopback address, "localhost" and "localnets" ACIs are no longer an issue for bind-dyndb-ldap. Allow them in our validators. -- Martin Kosek Senior Software Engineer - Identity Management Team Red Hat Inc. -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-304-allow-localhost-in-zone-acis.patch Type: text/x-patch Size: 1819 bytes Desc: not available URL: From tbabej at redhat.com Thu Sep 6 11:13:37 2012 From: tbabej at redhat.com (Tomas Babej) Date: Thu, 06 Sep 2012 13:13:37 +0200 Subject: [Freeipa-devel] [PATCH 0011] Make sure selinuxusemap behaves consistently to HBAC rule In-Reply-To: <50473DDF.8040303@redhat.com> References: <5044C8CE.20108@redhat.com> <50473DDF.8040303@redhat.com> Message-ID: <50488561.7090902@redhat.com> On 09/05/2012 01:56 PM, Martin Kosek wrote: > On 09/03/2012 05:12 PM, Tomas Babej wrote: >> Hi, >> >> Both selinuxusermap-add and selinuxusermap-mod commands now behave >> consistently in not allowing user/host category or user/host members >> and HBAC rule being set at the same time. Also adds a bunch of unit >> tests that check this behaviour. >> >> https://fedorahosted.org/freeipa/ticket/2983 >> >> Tomas >> > I found few issues with this patch: > > 1) Patch needs a rebase > > 2) Patch does not expect attributes to be set to None, i.e. to be left empty or > to be deleted, e.g.: > > # ipa selinuxusermap-add foo --selinuxuser=guest_u:s0 --usercat=all --hbacrule= > ipa: ERROR: HBAC rule and local members cannot both be set > > # ipa selinuxusermap-add foo --selinuxuser=guest_u:s0 --usercat=all > ---------------------------- > Added SELinux User Map "foo" > ---------------------------- > Rule name: foo > SELinux User: guest_u:s0 > User category: all > Enabled: TRUE > > # ipa selinuxusermap-mod foo --usercat= --hbacrule= > ipa: ERROR: HBAC rule and local members cannot both be set > > # ipa selinuxusermap-mod foo --usercat= > ------------------------------- > Modified SELinux User Map "foo" > ------------------------------- > Rule name: foo > SELinux User: guest_u:s0 > Enabled: TRUE > > # ipa selinuxusermap-mod foo --hbacrule=foo > ------------------------------- > Modified SELinux User Map "foo" > ------------------------------- > Rule name: foo > SELinux User: guest_u:s0 > HBAC Rule: foo > Enabled: TRUE > > # ipa selinuxusermap-mod foo --hbacrule= --usercat=all > ipa: ERROR: HBAC rule and local members cannot both be set > > All these validation failures are not valid. > > 3) Additionally, I think it would be more readable and less error prone that if > instead of this blob: > > + are_local_members_to_be_set = 'usercategory' in _entry_attrs or \ > + 'hostcategory' in _entry_attrs or \ > + 'memberuser' in _entry_attrs or \ > + 'memberhost' in _entry_attrs > > You would use something like that: > > are_local_members_to_be_set = any(attr in _entry_attrs > for attr in ('usercategory', > 'hostcategory', > 'memberuser', > 'memberhost')) > > Martin 1.) Done. 2.) Corrected. 3.) Fixed. Tomas -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-tbabej-0011-2-Make-sure-selinuxusemap-behaves-consistently-to-HBAC.patch Type: text/x-patch Size: 11800 bytes Desc: not available URL: From pvoborni at redhat.com Thu Sep 6 11:35:42 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Thu, 06 Sep 2012 13:35:42 +0200 Subject: [Freeipa-devel] [PATCH] 304 Allow localhost in zone ACIs In-Reply-To: <50487215.70600@redhat.com> References: <50487215.70600@redhat.com> Message-ID: <50488A8E.4030409@redhat.com> On 09/06/2012 11:51 AM, Martin Kosek wrote: > Loopback address, "localhost" and "localnets" ACIs are no longer > an issue for bind-dyndb-ldap. Allow them in our validators. > Martin's patch works and looks good - ACK. Attaching patch for Web UI part. -- Petr Vobornik -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pvoborni-0210-Allow-localhost-in-zone-ACIs-Web-UI.patch Type: text/x-patch Size: 2035 bytes Desc: not available URL: From sbose at redhat.com Thu Sep 6 11:39:47 2012 From: sbose at redhat.com (Sumit Bose) Date: Thu, 6 Sep 2012 13:39:47 +0200 Subject: [Freeipa-devel] [PATCHES] Various fixes for trust and range CLI Message-ID: <20120906113947.GD31872@localhost.localdomain> Hi, this series of patches touches couple of tickets related to the trust and (id)range CLI. I post them together because some of them depend on each other. I already rebased them on Martin's "Add range safety check for range_mod and range_del" patch which I'm currently reviewing. bye, Sumit -------------- next part -------------- From 2a87bd560b0a348430e20c59ad8c2e1657ad1d6d Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Thu, 23 Aug 2012 14:17:34 +0200 Subject: [PATCH 60/66] Rename range CLI to idrange --- API.txt | 146 ++++++++++----------- install/ui/Makefile.am | 2 +- install/ui/{range.js => idrange.js} | 62 ++++----- install/ui/index.html | 2 +- install/ui/jsl.conf | 2 +- .../test/data/{range_add.json => idrange_add.json} | 0 .../data/{range_find.json => idrange_find.json} | 0 ...nge_find_pkeys.json => idrange_find_pkeys.json} | 0 ...e_get_records.json => idrange_get_records.json} | 0 .../test/data/{range_mod.json => idrange_mod.json} | 0 .../data/{range_show.json => idrange_show.json} | 0 install/ui/test/data/ipa_init.json | 2 +- install/ui/test/data/ipa_init_commands.json | 12 +- install/ui/test/data/ipa_init_objects.json | 4 +- install/ui/webui.js | 2 +- ipalib/plugins/{range.py => idrange.py} | 28 ++-- ipalib/plugins/internal.py | 2 +- ipalib/plugins/trust.py | 4 +- tests/test_xmlrpc/test_range_plugin.py | 40 +++--- 19 Dateien ge?ndert, 154 Zeilen hinzugef?gt(+), 154 Zeilen entfernt(-) rename install/ui/{range.js => idrange.js} (64%) rename install/ui/test/data/{range_add.json => idrange_add.json} (100%) rename install/ui/test/data/{range_find.json => idrange_find.json} (100%) rename install/ui/test/data/{range_find_pkeys.json => idrange_find_pkeys.json} (100%) rename install/ui/test/data/{range_get_records.json => idrange_get_records.json} (100%) rename install/ui/test/data/{range_mod.json => idrange_mod.json} (100%) rename install/ui/test/data/{range_show.json => idrange_show.json} (100%) rename ipalib/plugins/{range.py => idrange.py} (95%) diff --git a/API.txt b/API.txt index aef12b7eb6e458d614c84ba20d782ef3154c09f0..f71d42c1ffc17d46b22ebca84868947d9fe376ba 100644 --- a/API.txt +++ b/API.txt @@ -1883,6 +1883,79 @@ output: Output('value', , None) command: i18n_messages args: 0,0,1 output: Output('messages', , None) +command: idrange_add +args: 1,11,3 +arg: Str('cn', attribute=True, cli_name='name', multivalue=False, primary_key=True, required=True) +option: Int('ipabaseid', attribute=True, cli_name='base_id', multivalue=False, required=True) +option: Int('ipaidrangesize', attribute=True, cli_name='range_size', multivalue=False, required=True) +option: Int('ipabaserid', attribute=True, cli_name='rid_base', multivalue=False, required=True) +option: Int('ipasecondarybaserid', attribute=True, cli_name='secondary_rid_base', multivalue=False, required=False) +option: Str('ipanttrusteddomainsid', attribute=True, cli_name='dom_sid', multivalue=False, required=False) +option: Str('iparangetype', attribute=True, cli_name='iparangetype', multivalue=False, required=False) +option: Str('setattr*', cli_name='setattr', exclude='webui') +option: Str('addattr*', cli_name='addattr', exclude='webui') +option: Flag('all', autofill=True, cli_name='all', default=False, exclude='webui') +option: Flag('raw', autofill=True, cli_name='raw', default=False, exclude='webui') +option: Str('version?', exclude='webui') +output: Output('summary', (, ), None) +output: Entry('result', , Gettext('A dictionary representing an LDAP entry', domain='ipa', localedir=None)) +output: Output('value', , None) +command: idrange_del +args: 1,1,3 +arg: Str('cn', attribute=True, cli_name='name', multivalue=True, primary_key=True, query=True, required=True) +option: Flag('continue', autofill=True, cli_name='continue', default=False) +output: Output('summary', (, ), None) +output: Output('result', , None) +output: Output('value', , None) +command: idrange_find +args: 1,13,4 +arg: Str('criteria?', noextrawhitespace=False) +option: Str('cn', attribute=True, autofill=False, cli_name='name', multivalue=False, primary_key=True, query=True, required=False) +option: Int('ipabaseid', attribute=True, autofill=False, cli_name='base_id', multivalue=False, query=True, required=False) +option: Int('ipaidrangesize', attribute=True, autofill=False, cli_name='range_size', multivalue=False, query=True, required=False) +option: Int('ipabaserid', attribute=True, autofill=False, cli_name='rid_base', multivalue=False, query=True, required=False) +option: Int('ipasecondarybaserid', attribute=True, autofill=False, cli_name='secondary_rid_base', multivalue=False, query=True, required=False) +option: Str('ipanttrusteddomainsid', attribute=True, autofill=False, cli_name='dom_sid', multivalue=False, query=True, required=False) +option: Str('iparangetype', attribute=True, autofill=False, cli_name='iparangetype', multivalue=False, query=True, required=False) +option: Int('timelimit?', autofill=False, minvalue=0) +option: Int('sizelimit?', autofill=False, minvalue=0) +option: Flag('all', autofill=True, cli_name='all', default=False, exclude='webui') +option: Flag('raw', autofill=True, cli_name='raw', default=False, exclude='webui') +option: Str('version?', exclude='webui') +option: Flag('pkey_only?', autofill=True, default=False) +output: Output('summary', (, ), None) +output: ListOfEntries('result', (, ), Gettext('A list of LDAP entries', domain='ipa', localedir=None)) +output: Output('count', , None) +output: Output('truncated', , None) +command: idrange_mod +args: 1,13,3 +arg: Str('cn', attribute=True, cli_name='name', multivalue=False, primary_key=True, query=True, required=True) +option: Int('ipabaseid', attribute=True, autofill=False, cli_name='base_id', multivalue=False, required=False) +option: Int('ipaidrangesize', attribute=True, autofill=False, cli_name='range_size', multivalue=False, required=False) +option: Int('ipabaserid', attribute=True, autofill=False, cli_name='rid_base', multivalue=False, required=False) +option: Int('ipasecondarybaserid', attribute=True, autofill=False, cli_name='secondary_rid_base', multivalue=False, required=False) +option: Str('ipanttrusteddomainsid', attribute=True, autofill=False, cli_name='dom_sid', multivalue=False, required=False) +option: Str('iparangetype', attribute=True, autofill=False, cli_name='iparangetype', multivalue=False, required=False) +option: Str('setattr*', cli_name='setattr', exclude='webui') +option: Str('addattr*', cli_name='addattr', exclude='webui') +option: Str('delattr*', cli_name='delattr', exclude='webui') +option: Flag('rights', autofill=True, default=False) +option: Flag('all', autofill=True, cli_name='all', default=False, exclude='webui') +option: Flag('raw', autofill=True, cli_name='raw', default=False, exclude='webui') +option: Str('version?', exclude='webui') +output: Output('summary', (, ), None) +output: Entry('result', , Gettext('A dictionary representing an LDAP entry', domain='ipa', localedir=None)) +output: Output('value', , None) +command: idrange_show +args: 1,4,3 +arg: Str('cn', attribute=True, cli_name='name', multivalue=False, primary_key=True, query=True, required=True) +option: Flag('rights', autofill=True, default=False) +option: Flag('all', autofill=True, cli_name='all', default=False, exclude='webui') +option: Flag('raw', autofill=True, cli_name='raw', default=False, exclude='webui') +option: Str('version?', exclude='webui') +output: Output('summary', (, ), None) +output: Entry('result', , Gettext('A dictionary representing an LDAP entry', domain='ipa', localedir=None)) +output: Output('value', , None) command: json_metadata args: 2,3,3 arg: Str('objname?') @@ -2376,79 +2449,6 @@ option: Str('version?', exclude='webui') output: Output('summary', (, ), None) output: Entry('result', , Gettext('A dictionary representing an LDAP entry', domain='ipa', localedir=None)) output: Output('value', , None) -command: range_add -args: 1,11,3 -arg: Str('cn', attribute=True, cli_name='name', multivalue=False, primary_key=True, required=True) -option: Int('ipabaseid', attribute=True, cli_name='base_id', multivalue=False, required=True) -option: Int('ipaidrangesize', attribute=True, cli_name='range_size', multivalue=False, required=True) -option: Int('ipabaserid', attribute=True, cli_name='rid_base', multivalue=False, required=True) -option: Int('ipasecondarybaserid', attribute=True, cli_name='secondary_rid_base', multivalue=False, required=False) -option: Str('ipanttrusteddomainsid', attribute=True, cli_name='dom_sid', multivalue=False, required=False) -option: Str('iparangetype', attribute=True, cli_name='iparangetype', multivalue=False, required=False) -option: Str('setattr*', cli_name='setattr', exclude='webui') -option: Str('addattr*', cli_name='addattr', exclude='webui') -option: Flag('all', autofill=True, cli_name='all', default=False, exclude='webui') -option: Flag('raw', autofill=True, cli_name='raw', default=False, exclude='webui') -option: Str('version?', exclude='webui') -output: Output('summary', (, ), None) -output: Entry('result', , Gettext('A dictionary representing an LDAP entry', domain='ipa', localedir=None)) -output: Output('value', , None) -command: range_del -args: 1,1,3 -arg: Str('cn', attribute=True, cli_name='name', multivalue=True, primary_key=True, query=True, required=True) -option: Flag('continue', autofill=True, cli_name='continue', default=False) -output: Output('summary', (, ), None) -output: Output('result', , None) -output: Output('value', , None) -command: range_find -args: 1,13,4 -arg: Str('criteria?', noextrawhitespace=False) -option: Str('cn', attribute=True, autofill=False, cli_name='name', multivalue=False, primary_key=True, query=True, required=False) -option: Int('ipabaseid', attribute=True, autofill=False, cli_name='base_id', multivalue=False, query=True, required=False) -option: Int('ipaidrangesize', attribute=True, autofill=False, cli_name='range_size', multivalue=False, query=True, required=False) -option: Int('ipabaserid', attribute=True, autofill=False, cli_name='rid_base', multivalue=False, query=True, required=False) -option: Int('ipasecondarybaserid', attribute=True, autofill=False, cli_name='secondary_rid_base', multivalue=False, query=True, required=False) -option: Str('ipanttrusteddomainsid', attribute=True, autofill=False, cli_name='dom_sid', multivalue=False, query=True, required=False) -option: Str('iparangetype', attribute=True, autofill=False, cli_name='iparangetype', multivalue=False, query=True, required=False) -option: Int('timelimit?', autofill=False, minvalue=0) -option: Int('sizelimit?', autofill=False, minvalue=0) -option: Flag('all', autofill=True, cli_name='all', default=False, exclude='webui') -option: Flag('raw', autofill=True, cli_name='raw', default=False, exclude='webui') -option: Str('version?', exclude='webui') -option: Flag('pkey_only?', autofill=True, default=False) -output: Output('summary', (, ), None) -output: ListOfEntries('result', (, ), Gettext('A list of LDAP entries', domain='ipa', localedir=None)) -output: Output('count', , None) -output: Output('truncated', , None) -command: range_mod -args: 1,13,3 -arg: Str('cn', attribute=True, cli_name='name', multivalue=False, primary_key=True, query=True, required=True) -option: Int('ipabaseid', attribute=True, autofill=False, cli_name='base_id', multivalue=False, required=False) -option: Int('ipaidrangesize', attribute=True, autofill=False, cli_name='range_size', multivalue=False, required=False) -option: Int('ipabaserid', attribute=True, autofill=False, cli_name='rid_base', multivalue=False, required=False) -option: Int('ipasecondarybaserid', attribute=True, autofill=False, cli_name='secondary_rid_base', multivalue=False, required=False) -option: Str('ipanttrusteddomainsid', attribute=True, autofill=False, cli_name='dom_sid', multivalue=False, required=False) -option: Str('iparangetype', attribute=True, autofill=False, cli_name='iparangetype', multivalue=False, required=False) -option: Str('setattr*', cli_name='setattr', exclude='webui') -option: Str('addattr*', cli_name='addattr', exclude='webui') -option: Str('delattr*', cli_name='delattr', exclude='webui') -option: Flag('rights', autofill=True, default=False) -option: Flag('all', autofill=True, cli_name='all', default=False, exclude='webui') -option: Flag('raw', autofill=True, cli_name='raw', default=False, exclude='webui') -option: Str('version?', exclude='webui') -output: Output('summary', (, ), None) -output: Entry('result', , Gettext('A dictionary representing an LDAP entry', domain='ipa', localedir=None)) -output: Output('value', , None) -command: range_show -args: 1,4,3 -arg: Str('cn', attribute=True, cli_name='name', multivalue=False, primary_key=True, query=True, required=True) -option: Flag('rights', autofill=True, default=False) -option: Flag('all', autofill=True, cli_name='all', default=False, exclude='webui') -option: Flag('raw', autofill=True, cli_name='raw', default=False, exclude='webui') -option: Str('version?', exclude='webui') -output: Output('summary', (, ), None) -output: Entry('result', , Gettext('A dictionary representing an LDAP entry', domain='ipa', localedir=None)) -output: Output('value', , None) command: role_add args: 1,6,3 arg: Str('cn', attribute=True, cli_name='name', multivalue=False, primary_key=True, required=True) diff --git a/install/ui/Makefile.am b/install/ui/Makefile.am index 841e210e27e209d8a2117aa29a27c77359d6c01b..1dc3216021e89754f08f6f2c2a58b260f7b0ce28 100644 --- a/install/ui/Makefile.am +++ b/install/ui/Makefile.am @@ -54,7 +54,7 @@ app_DATA = \ overpass_regular-web.ttf \ overpass_regular-web.woff \ policy.js \ - range.js \ + idrange.js \ reset_password.js \ reset_password.html \ rule.js \ diff --git a/install/ui/range.js b/install/ui/idrange.js similarity index 64% rename from install/ui/range.js rename to install/ui/idrange.js index ea9b47c982fff02ca988b8e1803fe95e1fd8cf2e..b2ce3169face41b2bd9327dfc016ba958436968d 100644 --- a/install/ui/range.js +++ b/install/ui/idrange.js @@ -22,9 +22,9 @@ /* REQUIRES: ipa.js, details.js, search.js, add.js, facet.js, entity.js */ -IPA.range = {}; +IPA.idrange = {}; -IPA.range.entity = function(spec) { +IPA.idrange.entity = function(spec) { var that = IPA.entity(spec); @@ -48,28 +48,28 @@ IPA.range.entity = function(spec) { 'iparangetype', { name: 'ipabaseid', - label: IPA.messages.objects.range.ipabaseid, - tooltip: IPA.get_entity_param('range', 'ipabaseid').label + label: IPA.messages.objects.idrange.ipabaseid, + tooltip: IPA.get_entity_param('idrange', 'ipabaseid').label }, { name: 'ipaidrangesize', - label: IPA.messages.objects.range.ipaidrangesize, - tooltip: IPA.get_entity_param('range', 'ipaidrangesize').label + label: IPA.messages.objects.idrange.ipaidrangesize, + tooltip: IPA.get_entity_param('idrange', 'ipaidrangesize').label }, { name: 'ipabaserid', - label: IPA.messages.objects.range.ipabaserid, - tooltip: IPA.get_entity_param('range', 'ipabaserid').label + label: IPA.messages.objects.idrange.ipabaserid, + tooltip: IPA.get_entity_param('idrange', 'ipabaserid').label }, { name: 'ipasecondarybaserid', - label: IPA.messages.objects.range.ipasecondarybaserid, - tooltip: IPA.get_entity_param('range', 'ipasecondarybaserid').label + label: IPA.messages.objects.idrange.ipasecondarybaserid, + tooltip: IPA.get_entity_param('idrange', 'ipasecondarybaserid').label }, { name: 'ipanttrusteddomainsid', - label: IPA.messages.objects.range.ipanttrusteddomainsid, - tooltip: IPA.get_entity_param('range', 'ipanttrusteddomainsid').label + label: IPA.messages.objects.idrange.ipanttrusteddomainsid, + tooltip: IPA.get_entity_param('idrange', 'ipanttrusteddomainsid').label } ] } @@ -79,43 +79,43 @@ IPA.range.entity = function(spec) { fields: [ { name: 'cn', - widget: 'range.cn' + widget: 'idrange.cn' }, { name: 'ipabaseid', - label: IPA.messages.objects.range.ipabaseid, - tooltip: IPA.get_entity_param('range', 'ipabaseid').label, - widget: 'range.ipabaseid' + label: IPA.messages.objects.idrange.ipabaseid, + tooltip: IPA.get_entity_param('idrange', 'ipabaseid').label, + widget: 'idrange.ipabaseid' }, { name: 'ipaidrangesize', - label: IPA.messages.objects.range.ipaidrangesize, - tooltip: IPA.get_entity_param('range', 'ipaidrangesize').label, - widget: 'range.ipaidrangesize' + label: IPA.messages.objects.idrange.ipaidrangesize, + tooltip: IPA.get_entity_param('idrange', 'ipaidrangesize').label, + widget: 'idrange.ipaidrangesize' }, { name: 'ipabaserid', - label: IPA.messages.objects.range.ipabaserid, - tooltip: IPA.get_entity_param('range', 'ipabaserid').label, - widget: 'range.ipabaserid' + label: IPA.messages.objects.idrange.ipabaserid, + tooltip: IPA.get_entity_param('idrange', 'ipabaserid').label, + widget: 'idrange.ipabaserid' }, { name: 'ipasecondarybaserid', - label: IPA.messages.objects.range.ipasecondarybaserid, - tooltip: IPA.get_entity_param('range', 'ipasecondarybaserid').label, + label: IPA.messages.objects.idrange.ipasecondarybaserid, + tooltip: IPA.get_entity_param('idrange', 'ipasecondarybaserid').label, widget: 'type.ipasecondarybaserid' }, { name: 'ipanttrusteddomainsid', - label: IPA.messages.objects.range.ipanttrusteddomainsid, - tooltip: IPA.get_entity_param('range', 'ipanttrusteddomainsid').label, + label: IPA.messages.objects.idrange.ipanttrusteddomainsid, + tooltip: IPA.get_entity_param('idrange', 'ipanttrusteddomainsid').label, widget: 'type.ipanttrusteddomainsid' } ], widgets: [ { type: 'details_table_section_nc', - name: 'range', + name: 'idrange', widgets: [ 'cn', 'ipabaseid', @@ -126,18 +126,18 @@ IPA.range.entity = function(spec) { { type: 'multiple_choice_section', name: 'type', - label: IPA.messages.objects.range.type, + label: IPA.messages.objects.idrange.type, choices: [ { name: 'local', - label: IPA.messages.objects.range.type_local, + label: IPA.messages.objects.idrange.type_local, fields: ['ipasecondarybaserid'], required: ['ipasecondarybaserid'], enabled: true }, { name: 'ad', - label: IPA.messages.objects.range.type_ad, + label: IPA.messages.objects.idrange.type_ad, fields: ['ipanttrusteddomainsid'], required: ['ipanttrusteddomainsid'] } @@ -159,4 +159,4 @@ IPA.range.entity = function(spec) { return that; }; -IPA.register('range', IPA.range.entity); +IPA.register('idrange', IPA.idrange.entity); diff --git a/install/ui/index.html b/install/ui/index.html index ab51b6d57b66158d0738697ef774f28b6efcab4a..ba86a521658c7d5ddc9f79485363548648d5b14d 100644 --- a/install/ui/index.html +++ b/install/ui/index.html @@ -53,7 +53,7 @@ - + diff --git a/install/ui/jsl.conf b/install/ui/jsl.conf index 5faeed9329de13729c75a97b23fcaa3ba62a1a6e..935566d4fa856aa59a2fbcb7078e73c8a99d30b7 100644 --- a/install/ui/jsl.conf +++ b/install/ui/jsl.conf @@ -157,7 +157,7 @@ +process dns.js +process automount.js +process automember.js -+process range.js ++process idrange.js +process trust.js +process webui.js +process login.js diff --git a/install/ui/test/data/range_add.json b/install/ui/test/data/idrange_add.json similarity index 100% rename from install/ui/test/data/range_add.json rename to install/ui/test/data/idrange_add.json diff --git a/install/ui/test/data/range_find.json b/install/ui/test/data/idrange_find.json similarity index 100% rename from install/ui/test/data/range_find.json rename to install/ui/test/data/idrange_find.json diff --git a/install/ui/test/data/range_find_pkeys.json b/install/ui/test/data/idrange_find_pkeys.json similarity index 100% rename from install/ui/test/data/range_find_pkeys.json rename to install/ui/test/data/idrange_find_pkeys.json diff --git a/install/ui/test/data/range_get_records.json b/install/ui/test/data/idrange_get_records.json similarity index 100% rename from install/ui/test/data/range_get_records.json rename to install/ui/test/data/idrange_get_records.json diff --git a/install/ui/test/data/range_mod.json b/install/ui/test/data/idrange_mod.json similarity index 100% rename from install/ui/test/data/range_mod.json rename to install/ui/test/data/idrange_mod.json diff --git a/install/ui/test/data/range_show.json b/install/ui/test/data/idrange_show.json similarity index 100% rename from install/ui/test/data/range_show.json rename to install/ui/test/data/idrange_show.json diff --git a/install/ui/test/data/ipa_init.json b/install/ui/test/data/ipa_init.json index 24364dafa9e996708b59946abb73c769c2096729..54ff66e33bf0148060e8788e41683e23522f0b80 100644 --- a/install/ui/test/data/ipa_init.json +++ b/install/ui/test/data/ipa_init.json @@ -368,7 +368,7 @@ "pwpolicy": { "identity": "Password Policy" }, - "range": { + "idrange": { "details": "Range Settings", "ipabaseid": "Base ID", "ipabaserid": "Primary RID base", diff --git a/install/ui/test/data/ipa_init_commands.json b/install/ui/test/data/ipa_init_commands.json index cfb2703f0a94823c5f0875ccbc3dfc3923446445..4237cc8a8d25cfddab25de979e21a1fcc4b82d1d 100644 --- a/install/ui/test/data/ipa_init_commands.json +++ b/install/ui/test/data/ipa_init_commands.json @@ -13708,7 +13708,7 @@ } ] }, - "range_add": { + "idrange_add": { "takes_args": [], "takes_options": [ { @@ -13797,7 +13797,7 @@ } ] }, - "range_del": { + "idrange_del": { "takes_args": [], "takes_options": [ { @@ -13811,7 +13811,7 @@ } ] }, - "range_find": { + "idrange_find": { "takes_args": [], "takes_options": [ { @@ -13941,7 +13941,7 @@ } ] }, - "range_mod": { + "idrange_mod": { "takes_args": [], "takes_options": [ { @@ -14045,7 +14045,7 @@ } ] }, - "range_show": { + "idrange_show": { "takes_args": [], "takes_options": [ { @@ -18180,4 +18180,4 @@ "methods": {}, "objects": {} } -} \ No newline at end of file +} diff --git a/install/ui/test/data/ipa_init_objects.json b/install/ui/test/data/ipa_init_objects.json index 02ac25f4d301323769173200a286bf0e1f07d22f..6458e4cdaa5c64ee0bd82df67508b002bc4446fb 100644 --- a/install/ui/test/data/ipa_init_objects.json +++ b/install/ui/test/data/ipa_init_objects.json @@ -5882,7 +5882,7 @@ ], "uuid_attribute": "" }, - "range": { + "idrange": { "aciattrs": [ "cn", "ipabaseid", @@ -7743,4 +7743,4 @@ } } } -} \ No newline at end of file +} diff --git a/install/ui/webui.js b/install/ui/webui.js index e91946011deb503bcd3cc3bd901f9ae35f927945..f381ab0bec7f48ff1bed204fbc5da7b6a5a5bd0a 100644 --- a/install/ui/webui.js +++ b/install/ui/webui.js @@ -84,7 +84,7 @@ IPA.admin_navigation = function(spec) { ]}, {entity: 'selfservice'}, {entity: 'delegation'}, - {entity: 'range'}, + {entity: 'idrange'}, {entity: 'trust'}, {entity: 'config'} ]}]; diff --git a/ipalib/plugins/range.py b/ipalib/plugins/idrange.py similarity index 95% rename from ipalib/plugins/range.py rename to ipalib/plugins/idrange.py index cc0c127531bb608fbe22b3332660e6412c28747d..23c8e0c3446d7c6c676134c8422a845e8d752820 100644 --- a/ipalib/plugins/range.py +++ b/ipalib/plugins/idrange.py @@ -31,7 +31,7 @@ __doc__ = _(""" Manage ID ranges """) -class range(LDAPObject): +class idrange(LDAPObject): """ Range object. """ @@ -46,8 +46,8 @@ class range(LDAPObject): 'ipasecondarybaserid', 'ipanttrusteddomainsid', 'iparangetype'] - label = _('Ranges') - label_singular = _('Range') + label = _('ID Ranges') + label_singular = _('ID Range') takes_params = ( Str('cn', @@ -143,7 +143,7 @@ class range(LDAPObject): error=_('range modification leaving objects with ID out ' 'of the defined range is not allowed')) -class range_add(LDAPCreate): +class idrange_add(LDAPCreate): __doc__ = _('Add new ID range.') msg_summary = _('Added ID range "%(value)s"') @@ -168,7 +168,7 @@ class range_add(LDAPCreate): self.obj.handle_iparangetype(entry_attrs, options, keep_objectclass=True) return dn -class range_del(LDAPDelete): +class idrange_del(LDAPDelete): __doc__ = _('Delete an ID range.') msg_summary = _('Deleted ID range "%(value)s"') @@ -185,7 +185,7 @@ class range_del(LDAPDelete): old_base_id, old_range_size, 0, 0) return dn -class range_find(LDAPSearch): +class idrange_find(LDAPSearch): __doc__ = _('Search for ranges.') msg_summary = ngettext( @@ -204,7 +204,7 @@ class range_find(LDAPSearch): self.obj.handle_iparangetype(entry, options) return truncated -class range_show(LDAPRetrieve): +class idrange_show(LDAPRetrieve): __doc__ = _('Display information about a range.') def pre_callback(self, ldap, dn, attrs_list, *keys, **options): @@ -217,7 +217,7 @@ class range_show(LDAPRetrieve): self.obj.handle_iparangetype(entry_attrs, options) return dn -class range_mod(LDAPUpdate): +class idrange_mod(LDAPUpdate): __doc__ = _('Modify ID range.') msg_summary = _('Modified ID range "%(value)s"') @@ -249,9 +249,9 @@ class range_mod(LDAPUpdate): self.obj.handle_iparangetype(entry_attrs, options) return dn -api.register(range) -api.register(range_add) -api.register(range_mod) -api.register(range_del) -api.register(range_find) -api.register(range_show) +api.register(idrange) +api.register(idrange_add) +api.register(idrange_mod) +api.register(idrange_del) +api.register(idrange_find) +api.register(idrange_show) diff --git a/ipalib/plugins/internal.py b/ipalib/plugins/internal.py index f449da4f4a856f67fd0490d3eb461db85c7cd40c..fbfcfea076fda0e02af05cf7c7f3e3bb3548343a 100644 --- a/ipalib/plugins/internal.py +++ b/ipalib/plugins/internal.py @@ -506,7 +506,7 @@ class i18n_messages(Command): "pwpolicy": { "identity": _("Password Policy"), }, - "range": { + "idrange": { "details": _("Range Settings"), "ipabaseid": _("Base ID"), "ipabaserid": _("Primary RID base"), diff --git a/ipalib/plugins/trust.py b/ipalib/plugins/trust.py index f19a0a874057860d0ae32f1cc2336bdc3accf6e5..5af5111d599fab28e0a62e4f1ea2f7976ab046b1 100644 --- a/ipalib/plugins/trust.py +++ b/ipalib/plugins/trust.py @@ -182,7 +182,7 @@ class trust_add(LDAPCreate): range_name = keys[-1].upper()+'_id_range' try: - old_range = api.Command['range_show'](range_name) + old_range = api.Command['idrange_show'](range_name) except errors.NotFound, e: old_range = None @@ -209,7 +209,7 @@ class trust_add(LDAPCreate): base_id = 200000 + (pysss_murmur.murmurhash3(dom_sid, len(dom_sid), 0xdeadbeef) % 10000) * 200000 try: - new_range = api.Command['range_add'](range_name, + new_range = api.Command['idrange_add'](range_name, ipabaseid=base_id, ipaidrangesize=options['range_size'], ipabaserid=0, diff --git a/tests/test_xmlrpc/test_range_plugin.py b/tests/test_xmlrpc/test_range_plugin.py index 4b7aa0893b9c77ebdec38e518bcf63ef88a6ce09..aab56e286ce04f6d3510eed0704b108c5cbe4bdc 100644 --- a/tests/test_xmlrpc/test_range_plugin.py +++ b/tests/test_xmlrpc/test_range_plugin.py @@ -18,7 +18,7 @@ # along with this program. If not, see . """ -Test the `ipalib/plugins/range.py` module, and XML-RPC in general. +Test the `ipalib/plugins/idrange.py` module, and XML-RPC in general. """ from ipalib import api, errors, _ @@ -38,15 +38,15 @@ group1_gid = 900100 class test_range(Declarative): cleanup_commands = [ - ('range_del', [testrange1], {}), + ('idrange_del', [testrange1], {}), ('user_del', [user1], {}), ('group_del', [group1], {}), ] tests = [ dict( - desc='Create range %r' % (testrange1), - command=('range_add', [testrange1], + desc='Create ID range %r' % (testrange1), + command=('idrange_add', [testrange1], dict(ipabaseid=testrange1_base_id, ipaidrangesize=testrange1_size, ipabaserid=1000, ipasecondarybaserid=20000)), expected=dict( @@ -67,8 +67,8 @@ class test_range(Declarative): ), dict( - desc='Retrieve range %r' % (testrange1), - command=('range_show', [testrange1], dict()), + desc='Retrieve ID range %r' % (testrange1), + command=('idrange_show', [testrange1], dict()), expected=dict( result=dict( dn=DN(('cn',testrange1),('cn','ranges'),('cn','etc'), @@ -87,7 +87,7 @@ class test_range(Declarative): dict( - desc='Create user %r in range %r' % (user1, testrange1), + desc='Create user %r in ID range %r' % (user1, testrange1), command=( 'user_add', [user1], dict(givenname=u'Test', sn=u'User1', uidnumber=user1_uid) @@ -124,7 +124,7 @@ class test_range(Declarative): dict( - desc='Create group %r in range %r' % (group1, testrange1), + desc='Create group %r in ID range %r' % (group1, testrange1), command=( 'group_add', [group1], dict(description=u'Test desc 1', gidnumber=group1_gid) @@ -145,8 +145,8 @@ class test_range(Declarative): dict( - desc='Try to modify range %r to get out bounds object #1' % (testrange1), - command=('range_mod', [testrange1], dict(ipabaseid=90001)), + desc='Try to modify ID range %r to get out bounds object #1' % (testrange1), + command=('idrange_mod', [testrange1], dict(ipabaseid=90001)), expected=errors.ValidationError(name='ipabaseid,ipaidrangesize', error=u'range modification leaving objects with ID out of the' u' defined range is not allowed'), @@ -154,8 +154,8 @@ class test_range(Declarative): dict( - desc='Try to modify range %r to get out bounds object #2' % (testrange1), - command=('range_mod', [testrange1], dict(ipaidrangesize=100)), + desc='Try to modify ID range %r to get out bounds object #2' % (testrange1), + command=('idrange_mod', [testrange1], dict(ipaidrangesize=100)), expected=errors.ValidationError(name='ipabaseid,ipaidrangesize', error=u'range modification leaving objects with ID out of the' u' defined range is not allowed'), @@ -163,8 +163,8 @@ class test_range(Declarative): dict( - desc='Try to modify range %r to get out bounds object #3' % (testrange1), - command=('range_mod', [testrange1], dict(ipabaseid=100, ipaidrangesize=100)), + desc='Try to modify ID range %r to get out bounds object #3' % (testrange1), + command=('idrange_mod', [testrange1], dict(ipabaseid=100, ipaidrangesize=100)), expected=errors.ValidationError(name='ipabaseid,ipaidrangesize', error=u'range modification leaving objects with ID out of the' u' defined range is not allowed'), @@ -172,8 +172,8 @@ class test_range(Declarative): dict( - desc='Modify range %r' % (testrange1), - command=('range_mod', [testrange1], dict(ipaidrangesize=90000)), + desc='Modify ID range %r' % (testrange1), + command=('idrange_mod', [testrange1], dict(ipaidrangesize=90000)), expected=dict( result=dict( cn=[testrange1], @@ -190,8 +190,8 @@ class test_range(Declarative): dict( - desc='Try to delete range %r with active IDs inside it' % testrange1, - command=('range_del', [testrange1], {}), + desc='Try to delete ID range %r with active IDs inside it' % testrange1, + command=('idrange_del', [testrange1], {}), expected=errors.ValidationError(name='ipabaseid,ipaidrangesize', error=u'range modification leaving objects with ID out of the' u' defined range is not allowed'), @@ -221,8 +221,8 @@ class test_range(Declarative): dict( - desc='Delete range %r' % testrange1, - command=('range_del', [testrange1], {}), + desc='Delete ID range %r' % testrange1, + command=('idrange_del', [testrange1], {}), expected=dict( result=dict(failed=u''), value=testrange1, -- 1.7.11.4 -------------- next part -------------- From c399d4930602748d8415e1154b5b1329596b6d12 Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Wed, 5 Sep 2012 13:21:04 +0200 Subject: [PATCH 61/66] IDRange CLI: allow to work without arguments Fixes https://fedorahosted.org/freeipa/ticket/2999 --- API.txt | 2 +- ipalib/plugins/idrange.py | 22 ++++++++++++++++------ 2 Dateien ge?ndert, 17 Zeilen hinzugef?gt(+), 7 Zeilen entfernt(-) diff --git a/API.txt b/API.txt index f71d42c1ffc17d46b22ebca84868947d9fe376ba..8982582786c6cbe47c884478c14e4724d7d6070a 100644 --- a/API.txt +++ b/API.txt @@ -1888,7 +1888,7 @@ args: 1,11,3 arg: Str('cn', attribute=True, cli_name='name', multivalue=False, primary_key=True, required=True) option: Int('ipabaseid', attribute=True, cli_name='base_id', multivalue=False, required=True) option: Int('ipaidrangesize', attribute=True, cli_name='range_size', multivalue=False, required=True) -option: Int('ipabaserid', attribute=True, cli_name='rid_base', multivalue=False, required=True) +option: Int('ipabaserid', attribute=True, cli_name='rid_base', multivalue=False, required=False) option: Int('ipasecondarybaserid', attribute=True, cli_name='secondary_rid_base', multivalue=False, required=False) option: Str('ipanttrusteddomainsid', attribute=True, cli_name='dom_sid', multivalue=False, required=False) option: Str('iparangetype', attribute=True, cli_name='iparangetype', multivalue=False, required=False) diff --git a/ipalib/plugins/idrange.py b/ipalib/plugins/idrange.py index 23c8e0c3446d7c6c676134c8422a845e8d752820..3231156bd8736db83912c73d990687ab4f4a8848 100644 --- a/ipalib/plugins/idrange.py +++ b/ipalib/plugins/idrange.py @@ -63,7 +63,7 @@ class idrange(LDAPObject): cli_name='range_size', label=_("Number of IDs in the range"), ), - Int('ipabaserid', + Int('ipabaserid?', cli_name='rid_base', label=_('First RID of the corresponding RID range'), ), @@ -150,15 +150,25 @@ class idrange_add(LDAPCreate): def pre_callback(self, ldap, dn, entry_attrs, attrs_list, *keys, **options): assert isinstance(dn, DN) - if ('ipanttrusteddomainsid' not in options and - 'ipasecondarybaserid' not in options): - raise errors.ValidationError(name=_('Range setup'), - error=_('Ranges for local domain ' \ - 'must have a secondary RID base')) if 'ipanttrusteddomainsid' in options: + if 'ipasecondarybaserid' in options: + raise errors.ValidationError(name=_('ID Range setup'), + error=_('Options dom_sid and secondary_rid_base cannot ' \ + 'be used together')) + + if 'ipabaserid' not in options: + raise errors.ValidationError(name=_('ID Range setup'), + error=_('Options dom_sid and rid_base must ' \ + 'be used together')) + entry_attrs['objectclass'].append('ipatrustedaddomainrange') else: + if (('ipasecondarybaserid' in options) != ('ipabaserid' in options)): + raise errors.ValidationError(name=_('ID Range setup'), + error=_('Options secondary_rid_base and rid_base must ' \ + 'be used together')) + entry_attrs['objectclass'].append('ipadomainidrange') return dn -- 1.7.11.4 -------------- next part -------------- From b37fad61093ff71b8ad12a49a6f9246fd3558074 Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Mon, 13 Aug 2012 17:07:37 +0200 Subject: [PATCH 62/66] IDRange CLI: Add documentation Fixes https://fedorahosted.org/freeipa/ticket/2969 --- ipalib/plugins/idrange.py | 129 +++++++++++++++++++++++++++++++++++++++++++++- 1 Datei ge?ndert, 127 Zeilen hinzugef?gt(+), 2 Zeilen entfernt(-) diff --git a/ipalib/plugins/idrange.py b/ipalib/plugins/idrange.py index 3231156bd8736db83912c73d990687ab4f4a8848..9604e1666afa38f5919324ca9dd11180c3c5b859 100644 --- a/ipalib/plugins/idrange.py +++ b/ipalib/plugins/idrange.py @@ -28,7 +28,113 @@ from ipapython.dn import DN __doc__ = _(""" -Manage ID ranges +ID ranges + +Manage ID ranges used to map Posix IDs to SIDs and back. + +There are two type of ID ranges which are both handled by this utility: + + - the ID ranges of the local domain + - the ID ranges of trusted remote domains + +Both types have the following attributes in common: + + - base-id: the first ID of the Posix ID range + - range-size: the size of the range + +With those two attributes a range object can reserve the Posix IDs starting +with base-id up to but not including base-id+range-size exclusively. + +Additionally an ID range of the local domain may set + - rid-base: the first RID(*) of the corresponding RID range + - secondary-rid-base: first RID of the secondary RID range + +and an ID range of a trusted domain must set + - rid-base: the first RID of the corresponding RID range + - dom_sid: domain SID of the trusted domain + + + +EXAMPLE: Add a new ID range for a trusted domain + +Since there might be more than one trusted domain the domain SID must be given +while creating the ID range. + + ipa range-add --base-id=1200000 --range-size=200000 --rid-base=0 \\ + --dom-sid=S-1-5-21-123-456-789 trusted_dom_range + +This ID range is then used by the IPA server and the SSSD IPA provider to +assign Posix UIDs to users from the trusted domain. + +If e.g a range for a trusted domain is configured with the following values: + base-id = 1200000 + range-size = 200000 + rid-base = 0 +the RIDs 0 to 199999 are mapped to the Posix ID from 1200000 to 13999999. So +RID 1000 <-> Posix ID 1201000 + + + +EXAMPLE: Add a new ID range for the local domain + +To create an ID range for the local domain it is not necessary to specify a +domain SID. But since it is possible that a user and a group can have the same +value as Posix ID a second RID interval is needed to handle conflicts. + + ipa range-add --base-id=1200000 --range-size=200000 --rid-base=1000 \\ + --secondary-rid-base=1000000 local_range + +The data from the ID ranges of the local domain are used by the IPA server +internally to assign SIDs to IPA users and groups. The SID will then be stored +in the user or group objects. + +If e.g. the ID range for the local domain is configured with the values from +the example above then a new user with the UID 1200007 will get the RID 1007. +If this RID is already used by a group the RID will be 1000007. This can only +happen if a user or a group object was created with a fixed ID because the +automatic assignment will not assign the same ID twice. Since there are only +users and groups sharing the same ID namespace it is sufficient to have only +one fallback range to handle conflicts. + +To find the Posix ID for a given RID from the local domain it has to be +checked first if the RID falls in the primary or secondary RID range and +the rid-base or the secondary-rid-base has to be subtracted, respectively, +and the base-id has to be added to get the Posix ID. + +Typically the creation of ID ranges happens behind the scenes and this CLI +must not be used at all. The ID range for the local domain will be created +during installation or upgrade from an older version. The ID range for a +trusted domain will be create together with the trust by 'ipa trust-add ...'. +The use cases for this CLI are + +USE CASES: + + Add an ID range from a transitively trusted domain + + If the trusted domain (A) trusts another domain (B) as well and this trust + is transitive 'ipa trust-add domain-A' will only create a range for + domain A. The ID range for domain B must be added manually. + + Add an additional ID range for the local domain + + If the ID range of the local domain is exhausted, i.e. no new IDs can be + assigned to Posix users or groups by the DNA plugin, a new range has to be + created to allow new users an groups to be added. (Currently there is no + connection between this range CLI and the DNA plugin, but a future version + might be able to modify the configuration of the DNS plugin as well) + +In general it is not necessary to modify or delete ID ranges. If there is no +other way to achieve a certain configuration than to modify or delete an ID +range it should be done with great care. Because UIDs are stored in the file +system and are used for access control it might be possible that users are +allowed to access files of other users if an ID range got deleted and reused +for a different domain. + +(*) The RID is typically the last integer of a user or group SID which follows +the domain SID. E.g. if the domain SID is S-1-5-21-123-456-789 and a user from +this domain has the SID S-1-5-21-123-456-789-1010 then 1010 id the RID of the +user. RIDs are unique in a domain, 32bit values and are used for users and +groups. """) class idrange(LDAPObject): @@ -144,7 +250,26 @@ class idrange(LDAPObject): 'of the defined range is not allowed')) class idrange_add(LDAPCreate): - __doc__ = _('Add new ID range.') + __doc__ = _(""" + Add new ID range. + + To add a new ID range you always have to specify + + --base-id + --range-size + + Additionally + + --rid-base + --econdary-rid-base + + may be given for a new ID range for the local domain while + + --rid-bas + --dom-sid + + must be given to add a new range for a trusted AD domain. + """) msg_summary = _('Added ID range "%(value)s"') -- 1.7.11.4 -------------- next part -------------- From e727bac7b0aaa2dad237a9cf0a25846dc3809d5b Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Wed, 5 Sep 2012 15:46:05 +0200 Subject: [PATCH 63/66] Do not create trust if murmur hash is not available and base-id not given Fixes https://fedorahosted.org/freeipa/ticket/3018 --- ipalib/plugins/trust.py | 11 +++++------ 1 Datei ge?ndert, 5 Zeilen hinzugef?gt(+), 6 Zeilen entfernt(-) diff --git a/ipalib/plugins/trust.py b/ipalib/plugins/trust.py index 5af5111d599fab28e0a62e4f1ea2f7976ab046b1..67f47932302a8eb0505dc58301034bc3fef8bccd 100644 --- a/ipalib/plugins/trust.py +++ b/ipalib/plugins/trust.py @@ -163,6 +163,11 @@ class trust_add(LDAPCreate): msg_summary = _('Added Active Directory trust for realm "%(value)s"') def execute(self, *keys, **options): + if not _murmur_installed and 'base_id' not in options: + raise errors.ValidationError(name=_('missing base_id'), + error=_('pysss_murmur is not available on the server ' \ + 'and no base-id is given.')) + if 'trust_type' in options: if options['trust_type'] == u'ad': result = self.execute_ad(*keys, **options) @@ -200,12 +205,6 @@ class trust_add(LDAPCreate): if 'base_id' in options: base_id = options['base_id'] else: - if not _murmur_installed: - raise errors.ValidationError(name=_('missing base_id'), - error=_('pysss_murmur is not available on the server ' \ - 'and no base_id is given, ' \ - 'ID range must be create manually')) - base_id = 200000 + (pysss_murmur.murmurhash3(dom_sid, len(dom_sid), 0xdeadbeef) % 10000) * 200000 try: -- 1.7.11.4 -------------- next part -------------- From 01f497ef461c3b10b41725255288ed3508412efa Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Wed, 5 Sep 2012 17:53:29 +0200 Subject: [PATCH 64/66] Trust CLI: Return more details when searching trusts Fixes https://fedorahosted.org/freeipa/ticket/2970 --- ipalib/plugins/trust.py | 13 +++++++++++++ 1 Datei ge?ndert, 13 Zeilen hinzugef?gt(+) diff --git a/ipalib/plugins/trust.py b/ipalib/plugins/trust.py index 67f47932302a8eb0505dc58301034bc3fef8bccd..b2f3e559737f8858ac669d3a6b738f1a61b35ddc 100644 --- a/ipalib/plugins/trust.py +++ b/ipalib/plugins/trust.py @@ -100,6 +100,8 @@ class trust(LDAPObject): 'ipanttrusttype', 'ipanttrustattributes', 'ipanttrustdirection', 'ipanttrustpartner', 'ipantauthtrustoutgoing', 'ipanttrustauthincoming', 'ipanttrustforesttrustinfo', 'ipanttrustposixoffset', 'ipantsupportedencryptiontypes' ] + search_display_attributes = ['cn', 'ipantflatname', + 'ipanttrusteddomainsid', 'ipanttrusttype' ] label = _('Trusts') label_singular = _('Trust') @@ -300,6 +302,7 @@ class trust_mod(LDAPUpdate): class trust_find(LDAPSearch): __doc__ = _('Search for trusts.') + has_output_params = LDAPSearch.has_output_params + trust_output_params msg_summary = ngettext( '%(count)d trust matched', '%(count)d trusts matched', 0 @@ -311,6 +314,16 @@ class trust_find(LDAPSearch): assert isinstance(base_dn, DN) return (filters, base_dn, ldap.SCOPE_SUBTREE) + def post_callback(self, ldap, entries, truncated, *args, **options): + if options.get('pkey_only', False): + return truncated + + for entry in entries: + (dn, attrs) = entry + attrs['trusttype'] = trust_type_string(attrs['ipanttrusttype'][0]) + + return truncated + class trust_show(LDAPRetrieve): __doc__ = _('Display information about a trust.') has_output_params = LDAPRetrieve.has_output_params + trust_output_params -- 1.7.11.4 -------------- next part -------------- From 83d488b83f962766970f5cae8a192265fc95825b Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Wed, 5 Sep 2012 18:50:10 +0200 Subject: [PATCH 65/66] Trust CLI: return more details of added trust Fixes: https://fedorahosted.org/freeipa/ticket/2971 --- ipalib/plugins/trust.py | 11 +++++++++++ 1 Datei ge?ndert, 11 Zeilen hinzugef?gt(+) diff --git a/ipalib/plugins/trust.py b/ipalib/plugins/trust.py index b2f3e559737f8858ac669d3a6b738f1a61b35ddc..48aa4027aecfba0ee0b5c084c90e91a2285fefd6 100644 --- a/ipalib/plugins/trust.py +++ b/ipalib/plugins/trust.py @@ -163,6 +163,7 @@ class trust_add(LDAPCreate): ) msg_summary = _('Added Active Directory trust for realm "%(value)s"') + has_output_params = LDAPCreate.has_output_params + trust_output_params def execute(self, *keys, **options): if not _murmur_installed and 'base_id' not in options: @@ -180,6 +181,16 @@ class trust_add(LDAPCreate): self.add_range(*keys, **options) + trust_filter = "cn=%s" % result['value'] + ldap = self.obj.backend + (trusts, truncated) = ldap.find_entries( + base_dn = DN(api.env.container_trusts, api.env.basedn), + filter = trust_filter) + + result['result'] = trusts[0][1] + result['result']['trusttype'] = [trust_type_string(result['result']['ipanttrusttype'][0])] + result['result']['trustdirection'] = [trust_direction_string(result['result']['ipanttrustdirection'][0])] + return result def add_range(self, *keys, **options): -- 1.7.11.4 -------------- next part -------------- From 8588d9c0d28af3d2fe9cbd23de02d556e27225fd Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Thu, 6 Sep 2012 09:48:07 +0200 Subject: [PATCH 66/66] Trust CLI: mark trust-mod for future use Fixes: https://fedorahosted.org/freeipa/ticket/2968 --- ipalib/plugins/trust.py | 7 ++++++- 1 Datei ge?ndert, 6 Zeilen hinzugef?gt(+), 1 Zeile entfernt(-) diff --git a/ipalib/plugins/trust.py b/ipalib/plugins/trust.py index 48aa4027aecfba0ee0b5c084c90e91a2285fefd6..074560dc27eb121b5035ba9a8260e5ab24b2b4b5 100644 --- a/ipalib/plugins/trust.py +++ b/ipalib/plugins/trust.py @@ -296,7 +296,12 @@ class trust_del(LDAPDelete): return result['result']['dn'] class trust_mod(LDAPUpdate): - __doc__ = _('Modify a trust.') + __doc__ = _(""" + Modify a trust (for future use). + + Currently only the default option to modify the LDAP attributes are + available. More specific options will be added in coming releases. + """) msg_summary = _('Modified trust "%(value)s"') -- 1.7.11.4 From mkosek at redhat.com Thu Sep 6 11:49:36 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 06 Sep 2012 13:49:36 +0200 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <50479491.1060200@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> <50479491.1060200@redhat.com> Message-ID: <50488DD0.1080507@redhat.com> On 09/05/2012 08:06 PM, Rob Crittenden wrote: > Rob Crittenden wrote: >> Martin Kosek wrote: >>> On 07/05/2012 08:39 PM, Rob Crittenden wrote: >>>> Martin Kosek wrote: >>>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: >>>>>> Deleting a replica can leave a replication vector (RUV) on the >>>>>> other servers. >>>>>> This can confuse things if the replica is re-added, and it also >>>>>> causes the >>>>>> server to calculate changes against a server that may no longer exist. >>>>>> >>>>>> 389-ds-base provides a new task that self-propogates itself to all >>>>>> available >>>>>> replicas to clean this RUV data. >>>>>> >>>>>> This patch will create this task at deletion time to hopefully >>>>>> clean things up. >>>>>> >>>>>> It isn't perfect. If any replica is down or unavailable at the time >>>>>> the >>>>>> cleanruv task fires, and then comes back up, the old RUV data may be >>>>>> re-propogated around. >>>>>> >>>>>> To make things easier in this case I've added two new commands to >>>>>> ipa-replica-manage. The first lists the replication ids of all the >>>>>> servers we >>>>>> have a RUV for. Using this you can call clean_ruv with the >>>>>> replication id of a >>>>>> server that no longer exists to try the cleanallruv step again. >>>>>> >>>>>> This is quite dangerous though. If you run cleanruv against a >>>>>> replica id that >>>>>> does exist it can cause a loss of data. I believe I've put in >>>>>> enough scary >>>>>> warnings about this. >>>>>> >>>>>> rob >>>>>> >>>>> >>>>> Good work there, this should make cleaning RUVs much easier than >>>>> with the >>>>> previous version. >>>>> >>>>> This is what I found during review: >>>>> >>>>> 1) list_ruv and clean_ruv command help in man is quite lost. I think >>>>> it would >>>>> help if we for example have all info for commands indented. This way >>>>> user could >>>>> simply over-look the new commands in the man page. >>>>> >>>>> >>>>> 2) I would rename new commands to clean-ruv and list-ruv to make them >>>>> consistent with the rest of the commands (re-initialize, force-sync). >>>>> >>>>> >>>>> 3) It would be nice to be able to run clean_ruv command in an >>>>> unattended way >>>>> (for better testing), i.e. respect --force option as we already do for >>>>> ipa-replica-manage del. This fix would aid test automation in the >>>>> future. >>>>> >>>>> >>>>> 4) (minor) The new question (and the del too) does not react too >>>>> well for >>>>> CTRL+D: >>>>> >>>>> # ipa-replica-manage clean_ruv 3 --force >>>>> Clean the Replication Update Vector for >>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>> >>>>> Cleaning the wrong replica ID will cause that server to no >>>>> longer replicate so it may miss updates while the process >>>>> is running. It would need to be re-initialized to maintain >>>>> consistency. Be very careful. >>>>> Continue to clean? [no]: unexpected error: >>>>> >>>>> >>>>> 5) Help for clean_ruv command without a required parameter is quite >>>>> confusing >>>>> as it reports that command is wrong and not the parameter: >>>>> >>>>> # ipa-replica-manage clean_ruv >>>>> Usage: ipa-replica-manage [options] >>>>> >>>>> ipa-replica-manage: error: must provide a command [clean_ruv | >>>>> force-sync | >>>>> disconnect | connect | del | re-initialize | list | list_ruv] >>>>> >>>>> It seems you just forgot to specify the error message in the command >>>>> definition >>>>> >>>>> >>>>> 6) When the remote replica is down, the clean_ruv command fails with an >>>>> unexpected error: >>>>> >>>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 >>>>> Clean the Replication Update Vector for >>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>> >>>>> Cleaning the wrong replica ID will cause that server to no >>>>> longer replicate so it may miss updates while the process >>>>> is running. It would need to be re-initialized to maintain >>>>> consistency. Be very careful. >>>>> Continue to clean? [no]: y >>>>> unexpected error: {'desc': 'Operations error'} >>>>> >>>>> >>>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: >>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>> cleanAllRUV_task: failed >>>>> to connect to repl agreement connection >>>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, >>>>> >>>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>> tree,cn=config), error 105 >>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>> cleanAllRUV_task: replica >>>>> (cn=meTovm-055.idm.lab. >>>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>> >>>>> >>>>> >>>>> tree, cn=config) has not been cleaned. You will need to rerun the >>>>> CLEANALLRUV task on this replica. >>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>> cleanAllRUV_task: Task >>>>> failed (1) >>>>> >>>>> In this case I think we should inform user that the command failed, >>>>> possibly >>>>> because of disconnected replicas and that they could enable the >>>>> replicas and >>>>> try again. >>>>> >>>>> >>>>> 7) (minor) "pass" is now redundant in replication.py: >>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>> + # We can't make the server we're removing read-only but >>>>> + # this isn't a show-stopper >>>>> + root_logger.debug("No permission to switch replica to >>>>> read-only, >>>>> continuing anyway") >>>>> + pass >>>>> >>>> >>>> I think this addresses everything. >>>> >>>> rob >>> >>> Thanks, almost there! I just found one more issue which needs to be fixed >>> before we push: >>> >>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>> Directory Manager password: >>> >>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing >>> removal >>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': "Can't >>> contact LDAP server"} >>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>> >>> There were issues removing a connection: %d format: a number is >>> required, not str >>> >>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': "Can't >>> contact LDAP server"} >>> >>> This is a traceback I retrieved: >>> Traceback (most recent call last): >>> File "/sbin/ipa-replica-manage", line 425, in del_master >>> del_link(realm, r, hostname, options.dirman_passwd, force=True) >>> File "/sbin/ipa-replica-manage", line 271, in del_link >>> repl1.cleanallruv(replica_id) >>> File >>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>> line 1094, in cleanallruv >>> root_logger.debug("Creating CLEANALLRUV task for replica id %d" % >>> replicaId) >>> >>> >>> The problem here is that you don't convert replica_id to int in this >>> part: >>> + replica_id = None >>> + if repl2: >>> + replica_id = repl2._get_replica_id(repl2.conn, None) >>> + else: >>> + servers = get_ruv(realm, replica1, dirman_passwd) >>> + for (netloc, rid) in servers: >>> + if netloc.startswith(replica2): >>> + replica_id = rid >>> + break >>> >>> Martin >>> >> >> Updated patch using new mechanism in 389-ds-base. This should more >> thoroughly clean out RUV data when a replica is being deleted, and >> provide for a way to delete RUV data afterwards too if necessary. >> >> rob > > Rebased patch > > rob > 0) As I wrote in a review for your patch 1041, changelog entry slipped elsewhere. 1) The following KeyboardInterrupt except class looks suspicious. I know why you have it there, but since it is generally a bad thing to do, some comment why it is needed would be useful. @@ -256,6 +263,17 @@ def del_link(realm, replica1, replica2, dirman_passwd, force=False): repl1.delete_agreement(replica2) repl1.delete_referral(replica2) + if type1 == replication.IPA_REPLICA: + if repl2: + ruv = repl2._get_replica_id(repl2.conn, None) + else: + ruv = get_ruv_by_host(realm, replica1, replica2, dirman_passwd) + + try: + repl1.cleanallruv(ruv) + except KeyboardInterrupt: + pass + Maybe you just wanted to do some cleanup and then "raise" again? 2) This is related to 1), but when some replica is down, "ipa-replica-manage del" may wait indefinitely when some remote replica is down, right? # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com Deleting a master is irreversible. To reconnect to the remote master you will need to prepare a new replica file and re-install. Continue to delete? [no]: y ipa: INFO: Setting agreement cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping tree,cn=config schedule to 2358-2359 0 to force synch ipa: INFO: Deleting schedule 2358-2359 0 from agreement cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping tree,cn=config ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica acquired successfully: Incremental update succeeded: start: 0: end: 0 Background task created to clean replication data ... after about a minute I hit CTRL+C ^CDeleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to 'vm-055.idm.lab.bos.redhat.com' Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: NS record does not contain 'vm-055.idm.lab.bos.redhat.com.' You may need to manually remove them from the tree I think it would be better to inform user that some remote replica is down or at least that we are waiting for the task to complete. Something like that: # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com ... Background task created to clean replication data Replication data clean up may take very long time if some replica is unreachable Hit CTRL+C to interrupt the wait ^C Clean up wait interrupted .... [continue with del] 3) (minor) When there is a cleanruv task running and you run "ipa-replica-manage del", there is a unexpected error message with duplicate task object in LDAP: # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com --force Unable to connect to replica vm-072.idm.lab.bos.redhat.com, forcing removal FAIL Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't contact LDAP server"} Forcing removal on 'vm-086.idm.lab.bos.redhat.com' There were issues removing a connection: This entry already exists <<<<<<<<< Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't contact LDAP server"} Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record does not contain 'vm-072.idm.lab.bos.redhat.com.' You may need to manually remove them from the tree I think it should be enough to just catch for "entry already exists" in cleanallruv function, and in such case print a relevant error message bail out. Thus, self.conn.checkTask(dn, dowait=True) would not be called too. 4) (minor): In make_readonly function, there is a redundant "pass" statement: + def make_readonly(self): + """ + Make the current replication agreement read-only. + """ + dn = DN(('cn', 'userRoot'), ('cn', 'ldbm database'), + ('cn', 'plugins'), ('cn', 'config')) + + mod = [(ldap.MOD_REPLACE, 'nsslapd-readonly', 'on')] + try: + self.conn.modify_s(dn, mod) + except ldap.INSUFFICIENT_ACCESS: + # We can't make the server we're removing read-only but + # this isn't a show-stopper + root_logger.debug("No permission to switch replica to read-only, continuing anyway") + pass <<<<<<<<<<<<<<< 5) In clean_ruv, I think allowing a --force option to bypass the user_input would be helpful (at least for test automation): + if not ipautil.user_input("Continue to clean?", False): + sys.exit("Aborted") Martin From mkosek at redhat.com Thu Sep 6 12:00:15 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 06 Sep 2012 14:00:15 +0200 Subject: [Freeipa-devel] [PATCH] 304 Allow localhost in zone ACIs In-Reply-To: <50488A8E.4030409@redhat.com> References: <50487215.70600@redhat.com> <50488A8E.4030409@redhat.com> Message-ID: <5048904F.6060802@redhat.com> On 09/06/2012 01:35 PM, Petr Vobornik wrote: > On 09/06/2012 11:51 AM, Martin Kosek wrote: >> Loopback address, "localhost" and "localnets" ACIs are no longer >> an issue for bind-dyndb-ldap. Allow them in our validators. >> > > Martin's patch works and looks good - ACK. > > > Attaching patch for Web UI part. Web UI validator works fine too, ACK. Pushed both patches to master, ipa-3-0. Martin From pviktori at redhat.com Thu Sep 6 12:10:06 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Thu, 06 Sep 2012 14:10:06 +0200 Subject: [Freeipa-devel] [PATCH] 0078 ipa-client-install: Obtain host TGT from one specific KDC Message-ID: <5048929E.7080605@redhat.com> When installing the client, we need to take extra case to only contact the one server we're installing against. Otherwise, in the real world, we might hit a server that hasn't replicated info about the client yet. This patch fixes a bug where kinit attempted to contact a KDC that didn't have the host principal yet. To reproduce: - Install a "master" and "replica" - Change the Kerberos DNS entries to only point to the replica: for REC_NAME in '_kerberos-master._tcp' '_kerberos-master._udp' '_kerberos._tcp' '_kerberos._udp' '_kpasswd._tcp' '_kpasswd._udp'; do ipa dnsrecord-mod $DOMAIN $REC_NAME --srv-rec="0 100 88 $REPLICA_HOSTNAME" done ipa dnsrecord-mod $DOMAIN _ldap._tcp --srv-rec="0 100 389 $MASTER_HOSTNAME" ipa dnsrecord-find $DOMAIN # check - Sever communication between the hosts to disable replication: (on master) iptables -A INPUT -j DROP -p all --source $REPLICA_IP - On client machine, put master as nameserver in /etc/resolv.conf & install client This will fail without the patch. Thanks to Petr Spacek, Simo, and Scott for helping to reproduce and explain the bug. I learned a lot. https://fedorahosted.org/freeipa/ticket/2982 -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0078-ipa-client-install-Obtain-host-TGT-from-one-specific.patch Type: text/x-patch Size: 5473 bytes Desc: not available URL: From mkosek at redhat.com Thu Sep 6 12:30:47 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 06 Sep 2012 14:30:47 +0200 Subject: [Freeipa-devel] [PATCH] 1053 support 389-ds posix-winsync plugin In-Reply-To: <5047965B.2080908@redhat.com> References: <5047950F.6080709@redhat.com> <5047965B.2080908@redhat.com> Message-ID: <50489777.5050700@redhat.com> On 09/05/2012 08:13 PM, Rich Megginson wrote: > On 09/05/2012 12:08 PM, Rob Crittenden wrote: >> Add support for the 389-ds posix winsync plugin. This plugin will sync the >> POSIX attributes from AD. We need to avoid trying to re-add them in our plugin. > ack I did a sanity check, that winsync replication still works, everything looks OK. Pushed to master, ipa-3-0. Martin From simo at redhat.com Thu Sep 6 12:35:31 2012 From: simo at redhat.com (Simo Sorce) Date: Thu, 06 Sep 2012 08:35:31 -0400 Subject: [Freeipa-devel] [PATCH] 1053 support 389-ds posix-winsync plugin In-Reply-To: <50489777.5050700@redhat.com> References: <5047950F.6080709@redhat.com> <5047965B.2080908@redhat.com> <50489777.5050700@redhat.com> Message-ID: <1346934931.1150.1440.camel@willson.li.ssimo.org> On Thu, 2012-09-06 at 14:30 +0200, Martin Kosek wrote: > On 09/05/2012 08:13 PM, Rich Megginson wrote: > > On 09/05/2012 12:08 PM, Rob Crittenden wrote: > >> Add support for the 389-ds posix winsync plugin. This plugin will sync the > >> POSIX attributes from AD. We need to avoid trying to re-add them in our plugin. > > ack > > I did a sanity check, that winsync replication still works, everything looks OK. > > Pushed to master, ipa-3-0. Is this plugin configurable in IPA ? >From the commit I can't tell if it is enabled by default or not. Simo. -- Simo Sorce * Red Hat, Inc * New York From rcritten at redhat.com Thu Sep 6 12:35:35 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 06 Sep 2012 08:35:35 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <1346893277.24731.60.camel@aleeredhat.laptop> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502B86F5.4070008@redhat.com> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> Message-ID: <50489897.2090601@redhat.com> Ade Lee wrote: > On Wed, 2012-09-05 at 16:20 -0400, Rob Crittenden wrote: >> Martin Kosek wrote: >>> On 08/31/2012 04:53 PM, Petr Viktorin wrote: >>>> On 08/28/2012 03:40 PM, Petr Viktorin wrote: >>>>> On 08/17/2012 06:04 PM, Ade Lee wrote: >>>>>> On Fri, 2012-08-17 at 09:34 -0400, Ade Lee wrote: >>>>>>> On Thu, 2012-08-16 at 18:45 +0200, Martin Kosek wrote: >>>>>>>> On 08/16/2012 01:28 PM, Ade Lee wrote: >>>>>>>>> Patch attached this time. I should know better than to do this in the >>>>>>>>> middle of the night .. >>>>>>>>> >>>>>>>>> On Thu, 2012-08-16 at 09:12 +0200, Martin Kosek wrote: >>>>>>>>>> On 08/16/2012 07:53 AM, Ade Lee wrote: >>>>>>>>>>> On Wed, 2012-08-15 at 23:41 -0400, Ade Lee wrote: >>>>>>>>>>>> On Wed, 2012-08-15 at 16:34 +0200, Martin Kosek wrote: >>>>>>>>>>>>> On 08/15/2012 03:54 PM, Ade Lee wrote: >>>>>>>>>>>>>> On Wed, 2012-08-15 at 13:24 +0200, Martin Kosek wrote: >>>>>>>>>>>>>>> On 08/08/2012 10:05 PM, Ade Lee wrote: >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Dogtag 10 is being released on f18, and has a number of >>>>>>>>>>>>>>>> changes that >>>>>>>>>>>>>>>> will affect IPA. In particular, the following changes will >>>>>>>>>>>>>>>> affect >>>>>>>>>>>>>>>> current IPA code. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> * The directory layout of the dogtag instance has changed. >>>>>>>>>>>>>>>> Instead of >>>>>>>>>>>>>>>> using separate tomcat instances to host different >>>>>>>>>>>>>>>> subsystems, the >>>>>>>>>>>>>>>> standard dogtag installation will allow one to install a CA. >>>>>>>>>>>>>>>> KRA, OCSP >>>>>>>>>>>>>>>> and TKS within the same instance. There have been >>>>>>>>>>>>>>>> corresponding changes >>>>>>>>>>>>>>>> in the directory layout, as well as the default instance name >>>>>>>>>>>>>>>> (pki-tomcat instead of pki-ca), and startup daemon >>>>>>>>>>>>>>>> (pki-tomcatd, instead >>>>>>>>>>>>>>>> of pki-cad, pki-krad etc.) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> * The default instance will use only four ports (HTTPS, >>>>>>>>>>>>>>>> HTTP, AJP and >>>>>>>>>>>>>>>> tomcat shutdown port) rather than the 6 previously used. >>>>>>>>>>>>>>>> The default >>>>>>>>>>>>>>>> ports will be changed to the standard tomcat ports. As >>>>>>>>>>>>>>>> these ports are >>>>>>>>>>>>>>>> local to the ipa server machine, this should not cause too much >>>>>>>>>>>>>>>> disruption. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> * There is a new single step installer written in python. >>>>>>>>>>>>>>>> (pkispawn/destroy) vs. pkicreate/pkisilent/pkiremove. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> * Dogtag 10 runs on tomcat7 - with a new corresponding >>>>>>>>>>>>>>>> version of >>>>>>>>>>>>>>>> tomcatjss. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The attached patch integrates all the above changes in IPA >>>>>>>>>>>>>>>> installation >>>>>>>>>>>>>>>> and maintenance code. Once the patch is applied, users will >>>>>>>>>>>>>>>> be able to: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1. run ipa-server-install to completion on f18 with dogtag 10. >>>>>>>>>>>>>>>> 2. install a new replica on f18 on dogtag 10. >>>>>>>>>>>>>>>> 3. upgrade an f17 machine with an existing IPA instance to >>>>>>>>>>>>>>>> f18/ dogtag >>>>>>>>>>>>>>>> 10 - and have that old-style dogtag instance continue to run >>>>>>>>>>>>>>>> correctly. >>>>>>>>>>>>>>>> This will require the installation of the latest version of >>>>>>>>>>>>>>>> tomcatjss as >>>>>>>>>>>>>>>> well as the installation of tomcat6. The old-style instance >>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>> continue to use tomcat6. >>>>>>>>>>>>>>>> 4. in addition, the new cert renewal code has been patched >>>>>>>>>>>>>>>> and should >>>>>>>>>>>>>>>> continue to work. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> What is not yet completed / supported: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1. Installation with an external CA is not yet completed in >>>>>>>>>>>>>>>> the new >>>>>>>>>>>>>>>> installer. We plan to complete this soon. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2. There is some IPA upgrade code that has not yet been touched >>>>>>>>>>>>>>>> (install/tools/ipa-upgradeconfig). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 3. A script needs to be written to allow admins to convert >>>>>>>>>>>>>>>> their >>>>>>>>>>>>>>>> old-style dogtag instances to new style instances, as well >>>>>>>>>>>>>>>> as code to >>>>>>>>>>>>>>>> periodically prompt admins to do this. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 4. Installation of old-style instances using >>>>>>>>>>>>>>>> pkicreate/pkisilent on >>>>>>>>>>>>>>>> dogtag 10 will no longer be supported, and will be disabled >>>>>>>>>>>>>>>> soon. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 5. The pki-selinux policy has been updated to reflect these >>>>>>>>>>>>>>>> changes, >>>>>>>>>>>>>>>> but is still in flux. In fact, it is our intention to place >>>>>>>>>>>>>>>> the dogtag >>>>>>>>>>>>>>>> selinux policy in the base selinux policy for f18. In the >>>>>>>>>>>>>>>> meantime, it >>>>>>>>>>>>>>>> may be necessary to run installs in permissive mode. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The dogtag 10 code will be released shortly into f18. Prior >>>>>>>>>>>>>>>> to that >>>>>>>>>>>>>>>> though, we have placed the new dogtag 10 and tomcatjss code >>>>>>>>>>>>>>>> in a >>>>>>>>>>>>>>>> developer repo that is located at >>>>>>>>>>>>>>>> http://nkinder.fedorapeople.org/dogtag-devel/ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Testing can be done on both f18 and f17 - although the >>>>>>>>>>>>>>>> target platform - >>>>>>>>>>>>>>>> and the only platform for which official builds will be >>>>>>>>>>>>>>>> created is f18. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Ade >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Ade, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks for the patch, I started with review and integration >>>>>>>>>>>>>>> tests (currently >>>>>>>>>>>>>>> running on Fedora 17 with Nathan's repo). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Installation on single master was smooth, it worked just >>>>>>>>>>>>>>> fine, even with >>>>>>>>>>>>>>> enforced SELinux, without any error - kudos to you and the >>>>>>>>>>>>>>> whole dogtag team. >>>>>>>>>>>>>>> The resulting logs and the structure of your log directory >>>>>>>>>>>>>>> seems improved. I >>>>>>>>>>>>>>> believe that the brand new Python installers will make it >>>>>>>>>>>>>>> easier to debug >>>>>>>>>>>>>>> issues with dogtag installation when they come. When I tried >>>>>>>>>>>>>>> our unit tests or >>>>>>>>>>>>>>> some simple cert operation, it worked fine as well. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Now the bad news, or rather few issues and suggestions I found: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1) As we already discussed on IRC, tomcat 7 was not pulled >>>>>>>>>>>>>>> automatically on >>>>>>>>>>>>>>> Fedora 17 when I updated pki-ca, you somewhere miss a Requires. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> We have a dogtag patch that is currently in review that will >>>>>>>>>>>>>> address >>>>>>>>>>>>>> this. Once this is in, tomcatjss >=7.0.0 will be required for >>>>>>>>>>>>>> f17+, >>>>>>>>>>>>>> rather than f18+ >>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2) I had installed IPA with dogtag10 on master. However, CA >>>>>>>>>>>>>>> installation on a >>>>>>>>>>>>>>> replica (ipa-ca-install) with dogtag9 failed - is this >>>>>>>>>>>>>>> expectable? >>>>>>>>>>>>>>> >>>>>>>>>>>>>> Yes. The current IPA patch is designed to work with dogtag 10 >>>>>>>>>>>>>> only, >>>>>>>>>>>>>> which will be officially available on f18+. So if you update >>>>>>>>>>>>>> to dogtag >>>>>>>>>>>>>> 10, you must have this patch and visa versa. We probably need >>>>>>>>>>>>>> to add >>>>>>>>>>>>>> the relevant requires to IPA to enforce this. >>>>>>>>>>>>>> >>>>>>>>>>>>>> If you have an existing dogtag 9 instance, and you upgrade to >>>>>>>>>>>>>> the new >>>>>>>>>>>>>> dogtag 10 and patched IPA, then that instance will continue to >>>>>>>>>>>>>> work. >>>>>>>>>>>>>> But any new instances would be created using dogtag 10. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> 3) I had installed IPA with dogtag10 on master. Replica had >>>>>>>>>>>>>>> dogtag10 as well >>>>>>>>>>>>>>> and I got the following error: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> # ipa-ca-install >>>>>>>>>>>>>>> /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg >>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>> Configuring certificate server: Estimated time 3 minutes 30 >>>>>>>>>>>>>>> seconds >>>>>>>>>>>>>>> [1/14]: creating certificate server user >>>>>>>>>>>>>>> [2/14]: configuring certificate server instance >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Your system may be partly configured. >>>>>>>>>>>>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Unexpected error - see /var/log/ipareplica-ca-install.log for >>>>>>>>>>>>>>> details: >>>>>>>>>>>>>>> IOError: [Errno 2] No such file or directory: >>>>>>>>>>>>>>> '/var/lib/pki/pki-tomcat/alias/ca_backup_keys.p12' >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Root cause: >>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>> File >>>>>>>>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", >>>>>>>>>>>>>>> line >>>>>>>>>>>>>>> 625, in __spawn_instance >>>>>>>>>>>>>>> "/root/cacert.p12") >>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>> >>>>>>>>>>>>>> I need to look into this. I had fixed ipa-replica-install, >>>>>>>>>>>>>> rather than >>>>>>>>>>>>>> ipa-ca-install to create replicas. I didn't know ipa-ca-install >>>>>>>>>>>>>> existed! Should not be too bad to fix though - most likely >>>>>>>>>>>>>> just need to >>>>>>>>>>>>>> move files to the right place. >>>>>>>>>>>>> >>>>>>>>>>>>> Ok, thanks! Btw. CA on replica can be either installed during >>>>>>>>>>>>> ipa-replica-install time (when --setup-ca option is passed, you >>>>>>>>>>>>> probably used >>>>>>>>>>>>> that one) and the aforementioned ipa-ca-install run after >>>>>>>>>>>>> ipa-replica-install. >>>>>>>>>>>>> >>>>>>>>>>>> I will be testing this out again. As ipa-ca-install uses the >>>>>>>>>>>> same code >>>>>>>>>>>> as ipa-replica-install, I would have expected it to work. Did >>>>>>>>>>>> you try >>>>>>>>>>>> it with selinux in permissive mode? >>>>>>>>>>>> >>>>>>>>>>> OK -- I tested this out and realized that between the time I >>>>>>>>>>> initially >>>>>>>>>>> tested this and your tests, we had changed a URL >>>>>>>>>>> from /ca/pki/installer/installToken to >>>>>>>>>>> /ca/rest/installer/installToken, >>>>>>>>>>> but I forgot to update ipa-pki-proxy.conf. >>>>>>>>>>> >>>>>>>>>>> The new patch has been rebased and changes the relevant patch. With >>>>>>>>>>> this, the CA replica installs correctly. >>>>>>>>>> >>>>>>>>>> Umh, where can I find the new rebased patch? :-) >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> There was one issue that I encountered. I have seen this once >>>>>>>>>>> before >>>>>>>>>>> and will need to investigate further. IPA sometimes restarts the >>>>>>>>>>> dogtag >>>>>>>>>>> instance by doing systemctl stop pki-tomcatd.target. and then >>>>>>>>>>> systemctl >>>>>>>>>>> start pki-tomcatd.target. I have noticed that occasionally the >>>>>>>>>>> start >>>>>>>>>>> invocation fails, while the stop and restart invocations succeed >>>>>>>>>>> just >>>>>>>>>>> fine. This makes absolutely no sense because restart is essentially >>>>>>>>>>> just stop and then start. >>>>>>>>>>> >>>>>>>>>>> I think this is actually a bug in systemd. If I reboot the >>>>>>>>>>> machine, it >>>>>>>>>>> all works perfectly. If we can't figure out whats going on with >>>>>>>>>>> systemd, we can always replace this start/stop with >>>>>>>>>>> starting/stopping >>>>>>>>>>> the service directly. (pki-tomcatd at pki-tomcat.service). That >>>>>>>>>>> seems to >>>>>>>>>>> work all the time with no problems. >>>>>>>>>>> >>>>>>>>>>> I suggest we leave this fix (if needed) for a separate patch. >>>>>>>>>> >>>>>>>>>> Ok, I will see if I hit this issue again. Btw. is it recommended >>>>>>>>>> in systemd to >>>>>>>>>> use target units this way? I thought that only service files are >>>>>>>>>> meant to be >>>>>>>>>> used for manipulation with a service. As you said yourself, using >>>>>>>>>> service unit >>>>>>>>>> file for restart worked every time. >>>>>>>>>> >>>>>>>>>> Martin >>>>>>>>> >>>>>>>> >>>>>>>> Now, I tested the rebased patch with VMs with permissive SELinux. >>>>>>>> IPA server >>>>>>>> install was OK, as always, but replica installation ended up with >>>>>>>> error. >>>>>>>> Although it got much further than before: >>>>>>>> >>>>>>>> # ipa-ca-install >>>>>>>> /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg >>>>>>>> ... >>>>>>>> [3/3]: restarting directory server >>>>>>>> done configuring pkids. >>>>>>>> Configuring certificate server: Estimated time 3 minutes 30 seconds >>>>>>>> [1/14]: creating certificate server user >>>>>>>> [2/14]: configuring certificate server instance >>>>>>>> [3/14]: disabling nonces >>>>>>>> [4/14]: importing CA chain to RA certificate database >>>>>>>> [5/14]: fixing RA database permissions >>>>>>>> [6/14]: setting up signing cert profile >>>>>>>> [7/14]: set up CRL publishing >>>>>>>> [8/14]: set certificate subject base >>>>>>>> [9/14]: enabling Subject Key Identifier >>>>>>>> [10/14]: configuring certificate server to start on boot >>>>>>>> [11/14]: configure certmonger for renewals >>>>>>>> [12/14]: configure clone certificate renewals >>>>>>>> [13/14]: configure Server-Cert certificate renewal >>>>>>>> [14/14]: Configure HTTP to proxy connections >>>>>>>> done configuring pki-tomcatd. >>>>>>>> Restarting the directory and certificate servers >>>>>>>> >>>>>>>> Your system may be partly configured. >>>>>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up. >>>>>>>> >>>>>>>> It seems like that CA on a replica did not start and so a check for >>>>>>>> port 8080 >>>>>>>> failed. Attached logs (pki-ca-logs.tgz) may provide more information >>>>>>>> to this issue. >>>>>>>> >>>>>>> This is the systemd issue I mentioned before. I will submit a patch >>>>>>> which will change the restart mechanism to restart the service and not >>>>>>> the target. >>>>>>>> >>>>>> >>>>>> Patch attached. This patch should be applied on top of the large patch >>>>>> (being reviewed). >>>>>> >>>>>>>> Now I am also testing upgrade from IPA+dogtag9 to PatchedIPA+dogtag10 >>>>>>>> (permissive SELinux). Now, the service was upgraded smoothly, CA >>>>>>>> service >>>>>>>> could be restarted without failure. However, certificate operations now >>>>>>>> did not work: >>>>>>>> >>>>>>>> # ipactl restart >>>>>>>> Restarting Directory Service >>>>>>>> Restarting KDC Service >>>>>>>> Restarting KPASSWD Service >>>>>>>> Restarting MEMCACHE Service >>>>>>>> Restarting HTTP Service >>>>>>>> Restarting CA Service >>>>>>>> # ipactl status >>>>>>>> Directory Service: RUNNING >>>>>>>> KDC Service: RUNNING >>>>>>>> KPASSWD Service: RUNNING >>>>>>>> MEMCACHE Service: RUNNING >>>>>>>> HTTP Service: RUNNING >>>>>>>> CA Service: RUNNING >>>>>>>> # ipa cert-show 1 >>>>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to >>>>>>>> communicate with CMS (Internal Server Error) >>>>>>>> >>>>>>>> Attaching logs for this case as well (ipa-ca-logs-upgrade.tgz). >>>>>>>> >>>>>>> The reason for this is that the old dogtag instances are missing: >>>>>>> 1. a new parameter in CS.cfg >>>>>>> 2. a couple of symlinks >>>>>>> >>>>>>> I plan to fix this in the dogtag code. Specifically, >>>>>>> 1. I will modify the dogtag code to provide a default value in the case >>>>>>> where the parameter does not exist. >>>>>>> 2. I plan to add a function to the dogtag startup scripts to check and >>>>>>> remake any required symlinks. >>>>>>> >>>>>>> Both of these changes are in the dogtag code, and do not affect this >>>>>>> patch. As a workaround, to confirm that the upgrade actually works, do >>>>>>> the following manual steps on your upgraded instance: >>>>>>> >>>>>>> 1. Add the following parameter to CS.cfg >>>>>>> pidDir=/var/run >>>>>>> >>>>>>> 2. Add the following links; >>>>>>> cd /var/lib/pki-ca/common/lib >>>>>>> ln -s /usr/share/java/apache-commons-codec.jar >>>>>>> apache-commons-codec.jar >>>>>>> ln -s /usr/share/java/log4j.jar log4j.jar >>>>>>> >>>>>>> 3 Restart the dogtag instance >>>>>>> >>>>>>>> HTH, >>>>>>>> Martin >>>>>>> >>>>>>> >>>>> >>>>> The attached patch conditionalizes the changes, so that IPA supports >>>>> both Dogtag versions. >>>>> >>>>> I didn't put it in platform code for two reasons >>>>> - One platform (f17) can have either Dogtag version installed >>>>> - I didn't want to conflict with Timo Aaltonen's "platformizing" work. >>>>> According to his WIP patches, he plans to move the whole dogtag module >>>>> into platform code. >>>>> >>>>> I used the existence of /usr/bin/pkispawn as a test for Dogtag 10. If >>>>> you know a better way, please comment. >>>>> >>>>> The dogtag_version option is now always added to >>>>> >>>>> I've reverted the changes to install/ui/test/data/ipa_init.json, so >>>>> you'll have to change the ports manually when testing the UI with Dogtag >>>>> 10. >>>>> >>>>> >>>>> >>>> >>>> Attaching a patch with fixed ipa-pki-proxy.conf, as discussed on IRC. >>>> >>> >>> This approach looks good. I did not hit any regression on F17 with dogtag9, or >>> clean installs of IPA+dogtag10. I think we are getting close to pushing this code. >>> >>> The only issues I hit so far are for upgraded dogtag9 instances (on F17): >>> >>> 1) The service did not start for me. There were some SELinux AVCs and even when >>> I disabled SELinux the instance still did not start (logs attached). >>> >>> 2) Uninstall was also not clean, we leave some dogtag installation states for >>> upgraded dogtag9 instance: >>> >>> # ipa-server-install --uninstall --unattended >>> Shutting down all IPA services >>> Removing IPA client configuration >>> Unconfiguring ntpd >>> Unconfiguring CA directory server >>> Unconfiguring web server >>> Unconfiguring krb5kdc >>> Unconfiguring kadmin >>> Unconfiguring directory server >>> Unconfiguring ipa_memcached >>> ipa : ERROR Some installation state for pki-cad has not been >>> restored, see /var/lib/ipa/sysrestore/sysrestore.state >>> >>> /var/lib/ipa/sysrestore/sysrestore.state: >>> [pki-cad] >>> enabled = False >> >> Ade is working on a new build to address the dogtag upgrade issues. >> >> The left over state should be removed when we drop the separate instance >> in the dogtag 9 -> 10 upgrade. I'm not completely sure reading the patch >> who actually does this part, is this handled by dogtag? >> >> rob >> > > The build is available on the developer repo. Just do a yum update. > > As for this leftover state, this is an ipa file and such is never > handled by dogtag. It must be handled somewhere within the ipa code. Well, the question is, what handles the dogtag 9 -> dogtag 10 upgrade? If IPA isn't involved then it has no chance to remove this state. rob From mkosek at redhat.com Thu Sep 6 12:45:22 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 06 Sep 2012 14:45:22 +0200 Subject: [Freeipa-devel] [PATCH] 1046 add e-mail by default In-Reply-To: <5037BFEB.8050602@redhat.com> References: <5037BFEB.8050602@redhat.com> Message-ID: <50489AE2.7020709@redhat.com> On 08/24/2012 07:54 PM, Rob Crittenden wrote: > We weren't automatically creating the mail attribute despite having the default > e-mail domain. This patch will add it to all new users. > > To disable creating this set the default e-mail domain to empty in ipa config. > > rob > 1) Patch needs a rebase 2) There are 2 test cases where new default mail attribute was not added: ====================================================================== FAIL: test_user[34]: user_find: Search for "tuser2" with manager "tuser1" ---------------------------------------------------------------------- ... extra keys = ['mail'] ... ====================================================================== FAIL: test_user[75]: user_add: Create 2nd admin user "admin2" ---------------------------------------------------------------------- ... extra keys = ['mail'] ... 3) Some code could be simplified: This: + if 'ipadefaultemaildomain' in config: + defaultdomain = config['ipadefaultemaildomain'][0] + else: + defaultdomain = None To this: defaultdomain = config.get('ipadefaultemaildomain', [None])[0] This: if m.find('@') == -1 ... To this: if '@' not in m ... IMHO, it is more readable than the find method. 3) When default e-mail domain is removed from config, users cannot be added any more when e-mail is not explicitly specified: # ipa config-mod --emaildomain= Maximum username length: 32 Home directory base: /home Default shell: /bin/sh Default users group: ipausers Search time limit: 2 Search size limit: 100 User search fields: uid,givenname,sn,telephonenumber,ou,title Group search fields: cn,description Enable migration mode: FALSE Certificate Subject base: O=IDM.LAB.BOS.REDHAT.COM Password Expiration Notification (days): 4 Password plugin features: AllowNThash SELinux user map order: guest_u:s0$xguest_u:s0$user_u:s0-s0:c0.c1023$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023 Default SELinux user: guest_u:s0 PAC type: MS-PAC # ipa user-add --first=Foo --last=Bar fbar ipa: ERROR: invalid 'email': invalid e-mail format: fbar Martin From pviktori at redhat.com Thu Sep 6 12:47:04 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Thu, 06 Sep 2012 14:47:04 +0200 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <50489897.2090601@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502B86F5.4070008@redhat.com> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> Message-ID: <50489B48.70503@redhat.com> On 09/06/2012 02:35 PM, Rob Crittenden wrote: > Ade Lee wrote: >> On Wed, 2012-09-05 at 16:20 -0400, Rob Crittenden wrote: >>> Martin Kosek wrote: >>>> On 08/31/2012 04:53 PM, Petr Viktorin wrote: >>>>> On 08/28/2012 03:40 PM, Petr Viktorin wrote: >>>>>> On 08/17/2012 06:04 PM, Ade Lee wrote: >>>>>>> On Fri, 2012-08-17 at 09:34 -0400, Ade Lee wrote: >>>>>>>> On Thu, 2012-08-16 at 18:45 +0200, Martin Kosek wrote: >>>>>>>>> On 08/16/2012 01:28 PM, Ade Lee wrote: >>>>>>>>>> Patch attached this time. I should know better than to do >>>>>>>>>> this in the >>>>>>>>>> middle of the night .. >>>>>>>>>> >>>>>>>>>> On Thu, 2012-08-16 at 09:12 +0200, Martin Kosek wrote: >>>>>>>>>>> On 08/16/2012 07:53 AM, Ade Lee wrote: >>>>>>>>>>>> On Wed, 2012-08-15 at 23:41 -0400, Ade Lee wrote: >>>>>>>>>>>>> On Wed, 2012-08-15 at 16:34 +0200, Martin Kosek wrote: >>>>>>>>>>>>>> On 08/15/2012 03:54 PM, Ade Lee wrote: >>>>>>>>>>>>>>> On Wed, 2012-08-15 at 13:24 +0200, Martin Kosek wrote: >>>>>>>>>>>>>>>> On 08/08/2012 10:05 PM, Ade Lee wrote: >>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Dogtag 10 is being released on f18, and has a number of >>>>>>>>>>>>>>>>> changes that >>>>>>>>>>>>>>>>> will affect IPA. In particular, the following changes >>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>> affect >>>>>>>>>>>>>>>>> current IPA code. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> * The directory layout of the dogtag instance has changed. >>>>>>>>>>>>>>>>> Instead of >>>>>>>>>>>>>>>>> using separate tomcat instances to host different >>>>>>>>>>>>>>>>> subsystems, the >>>>>>>>>>>>>>>>> standard dogtag installation will allow one to install >>>>>>>>>>>>>>>>> a CA. >>>>>>>>>>>>>>>>> KRA, OCSP >>>>>>>>>>>>>>>>> and TKS within the same instance. There have been >>>>>>>>>>>>>>>>> corresponding changes >>>>>>>>>>>>>>>>> in the directory layout, as well as the default >>>>>>>>>>>>>>>>> instance name >>>>>>>>>>>>>>>>> (pki-tomcat instead of pki-ca), and startup daemon >>>>>>>>>>>>>>>>> (pki-tomcatd, instead >>>>>>>>>>>>>>>>> of pki-cad, pki-krad etc.) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> * The default instance will use only four ports (HTTPS, >>>>>>>>>>>>>>>>> HTTP, AJP and >>>>>>>>>>>>>>>>> tomcat shutdown port) rather than the 6 previously used. >>>>>>>>>>>>>>>>> The default >>>>>>>>>>>>>>>>> ports will be changed to the standard tomcat ports. As >>>>>>>>>>>>>>>>> these ports are >>>>>>>>>>>>>>>>> local to the ipa server machine, this should not cause >>>>>>>>>>>>>>>>> too much >>>>>>>>>>>>>>>>> disruption. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> * There is a new single step installer written in python. >>>>>>>>>>>>>>>>> (pkispawn/destroy) vs. pkicreate/pkisilent/pkiremove. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> * Dogtag 10 runs on tomcat7 - with a new corresponding >>>>>>>>>>>>>>>>> version of >>>>>>>>>>>>>>>>> tomcatjss. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The attached patch integrates all the above changes in IPA >>>>>>>>>>>>>>>>> installation >>>>>>>>>>>>>>>>> and maintenance code. Once the patch is applied, users >>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>> be able to: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1. run ipa-server-install to completion on f18 with >>>>>>>>>>>>>>>>> dogtag 10. >>>>>>>>>>>>>>>>> 2. install a new replica on f18 on dogtag 10. >>>>>>>>>>>>>>>>> 3. upgrade an f17 machine with an existing IPA instance to >>>>>>>>>>>>>>>>> f18/ dogtag >>>>>>>>>>>>>>>>> 10 - and have that old-style dogtag instance continue >>>>>>>>>>>>>>>>> to run >>>>>>>>>>>>>>>>> correctly. >>>>>>>>>>>>>>>>> This will require the installation of the latest >>>>>>>>>>>>>>>>> version of >>>>>>>>>>>>>>>>> tomcatjss as >>>>>>>>>>>>>>>>> well as the installation of tomcat6. The old-style >>>>>>>>>>>>>>>>> instance >>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>> continue to use tomcat6. >>>>>>>>>>>>>>>>> 4. in addition, the new cert renewal code has been patched >>>>>>>>>>>>>>>>> and should >>>>>>>>>>>>>>>>> continue to work. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> What is not yet completed / supported: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1. Installation with an external CA is not yet >>>>>>>>>>>>>>>>> completed in >>>>>>>>>>>>>>>>> the new >>>>>>>>>>>>>>>>> installer. We plan to complete this soon. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2. There is some IPA upgrade code that has not yet been >>>>>>>>>>>>>>>>> touched >>>>>>>>>>>>>>>>> (install/tools/ipa-upgradeconfig). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 3. A script needs to be written to allow admins to convert >>>>>>>>>>>>>>>>> their >>>>>>>>>>>>>>>>> old-style dogtag instances to new style instances, as well >>>>>>>>>>>>>>>>> as code to >>>>>>>>>>>>>>>>> periodically prompt admins to do this. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 4. Installation of old-style instances using >>>>>>>>>>>>>>>>> pkicreate/pkisilent on >>>>>>>>>>>>>>>>> dogtag 10 will no longer be supported, and will be >>>>>>>>>>>>>>>>> disabled >>>>>>>>>>>>>>>>> soon. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 5. The pki-selinux policy has been updated to reflect >>>>>>>>>>>>>>>>> these >>>>>>>>>>>>>>>>> changes, >>>>>>>>>>>>>>>>> but is still in flux. In fact, it is our intention to >>>>>>>>>>>>>>>>> place >>>>>>>>>>>>>>>>> the dogtag >>>>>>>>>>>>>>>>> selinux policy in the base selinux policy for f18. In the >>>>>>>>>>>>>>>>> meantime, it >>>>>>>>>>>>>>>>> may be necessary to run installs in permissive mode. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The dogtag 10 code will be released shortly into f18. >>>>>>>>>>>>>>>>> Prior >>>>>>>>>>>>>>>>> to that >>>>>>>>>>>>>>>>> though, we have placed the new dogtag 10 and tomcatjss >>>>>>>>>>>>>>>>> code >>>>>>>>>>>>>>>>> in a >>>>>>>>>>>>>>>>> developer repo that is located at >>>>>>>>>>>>>>>>> http://nkinder.fedorapeople.org/dogtag-devel/ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Testing can be done on both f18 and f17 - although the >>>>>>>>>>>>>>>>> target platform - >>>>>>>>>>>>>>>>> and the only platform for which official builds will be >>>>>>>>>>>>>>>>> created is f18. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> Ade >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Ade, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks for the patch, I started with review and integration >>>>>>>>>>>>>>>> tests (currently >>>>>>>>>>>>>>>> running on Fedora 17 with Nathan's repo). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Installation on single master was smooth, it worked just >>>>>>>>>>>>>>>> fine, even with >>>>>>>>>>>>>>>> enforced SELinux, without any error - kudos to you and the >>>>>>>>>>>>>>>> whole dogtag team. >>>>>>>>>>>>>>>> The resulting logs and the structure of your log directory >>>>>>>>>>>>>>>> seems improved. I >>>>>>>>>>>>>>>> believe that the brand new Python installers will make it >>>>>>>>>>>>>>>> easier to debug >>>>>>>>>>>>>>>> issues with dogtag installation when they come. When I >>>>>>>>>>>>>>>> tried >>>>>>>>>>>>>>>> our unit tests or >>>>>>>>>>>>>>>> some simple cert operation, it worked fine as well. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Now the bad news, or rather few issues and suggestions I >>>>>>>>>>>>>>>> found: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1) As we already discussed on IRC, tomcat 7 was not pulled >>>>>>>>>>>>>>>> automatically on >>>>>>>>>>>>>>>> Fedora 17 when I updated pki-ca, you somewhere miss a >>>>>>>>>>>>>>>> Requires. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We have a dogtag patch that is currently in review that will >>>>>>>>>>>>>>> address >>>>>>>>>>>>>>> this. Once this is in, tomcatjss >=7.0.0 will be >>>>>>>>>>>>>>> required for >>>>>>>>>>>>>>> f17+, >>>>>>>>>>>>>>> rather than f18+ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2) I had installed IPA with dogtag10 on master. However, CA >>>>>>>>>>>>>>>> installation on a >>>>>>>>>>>>>>>> replica (ipa-ca-install) with dogtag9 failed - is this >>>>>>>>>>>>>>>> expectable? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yes. The current IPA patch is designed to work with >>>>>>>>>>>>>>> dogtag 10 >>>>>>>>>>>>>>> only, >>>>>>>>>>>>>>> which will be officially available on f18+. So if you >>>>>>>>>>>>>>> update >>>>>>>>>>>>>>> to dogtag >>>>>>>>>>>>>>> 10, you must have this patch and visa versa. We probably >>>>>>>>>>>>>>> need >>>>>>>>>>>>>>> to add >>>>>>>>>>>>>>> the relevant requires to IPA to enforce this. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If you have an existing dogtag 9 instance, and you >>>>>>>>>>>>>>> upgrade to >>>>>>>>>>>>>>> the new >>>>>>>>>>>>>>> dogtag 10 and patched IPA, then that instance will >>>>>>>>>>>>>>> continue to >>>>>>>>>>>>>>> work. >>>>>>>>>>>>>>> But any new instances would be created using dogtag 10. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 3) I had installed IPA with dogtag10 on master. Replica had >>>>>>>>>>>>>>>> dogtag10 as well >>>>>>>>>>>>>>>> and I got the following error: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> # ipa-ca-install >>>>>>>>>>>>>>>> /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg >>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>> Configuring certificate server: Estimated time 3 minutes 30 >>>>>>>>>>>>>>>> seconds >>>>>>>>>>>>>>>> [1/14]: creating certificate server user >>>>>>>>>>>>>>>> [2/14]: configuring certificate server instance >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Your system may be partly configured. >>>>>>>>>>>>>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Unexpected error - see >>>>>>>>>>>>>>>> /var/log/ipareplica-ca-install.log for >>>>>>>>>>>>>>>> details: >>>>>>>>>>>>>>>> IOError: [Errno 2] No such file or directory: >>>>>>>>>>>>>>>> '/var/lib/pki/pki-tomcat/alias/ca_backup_keys.p12' >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Root cause: >>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>> File >>>>>>>>>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> line >>>>>>>>>>>>>>>> 625, in __spawn_instance >>>>>>>>>>>>>>>> "/root/cacert.p12") >>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I need to look into this. I had fixed ipa-replica-install, >>>>>>>>>>>>>>> rather than >>>>>>>>>>>>>>> ipa-ca-install to create replicas. I didn't know >>>>>>>>>>>>>>> ipa-ca-install >>>>>>>>>>>>>>> existed! Should not be too bad to fix though - most likely >>>>>>>>>>>>>>> just need to >>>>>>>>>>>>>>> move files to the right place. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ok, thanks! Btw. CA on replica can be either installed during >>>>>>>>>>>>>> ipa-replica-install time (when --setup-ca option is >>>>>>>>>>>>>> passed, you >>>>>>>>>>>>>> probably used >>>>>>>>>>>>>> that one) and the aforementioned ipa-ca-install run after >>>>>>>>>>>>>> ipa-replica-install. >>>>>>>>>>>>>> >>>>>>>>>>>>> I will be testing this out again. As ipa-ca-install uses the >>>>>>>>>>>>> same code >>>>>>>>>>>>> as ipa-replica-install, I would have expected it to work. Did >>>>>>>>>>>>> you try >>>>>>>>>>>>> it with selinux in permissive mode? >>>>>>>>>>>>> >>>>>>>>>>>> OK -- I tested this out and realized that between the time I >>>>>>>>>>>> initially >>>>>>>>>>>> tested this and your tests, we had changed a URL >>>>>>>>>>>> from /ca/pki/installer/installToken to >>>>>>>>>>>> /ca/rest/installer/installToken, >>>>>>>>>>>> but I forgot to update ipa-pki-proxy.conf. >>>>>>>>>>>> >>>>>>>>>>>> The new patch has been rebased and changes the relevant >>>>>>>>>>>> patch. With >>>>>>>>>>>> this, the CA replica installs correctly. >>>>>>>>>>> >>>>>>>>>>> Umh, where can I find the new rebased patch? :-) >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> There was one issue that I encountered. I have seen this once >>>>>>>>>>>> before >>>>>>>>>>>> and will need to investigate further. IPA sometimes >>>>>>>>>>>> restarts the >>>>>>>>>>>> dogtag >>>>>>>>>>>> instance by doing systemctl stop pki-tomcatd.target. and then >>>>>>>>>>>> systemctl >>>>>>>>>>>> start pki-tomcatd.target. I have noticed that occasionally >>>>>>>>>>>> the >>>>>>>>>>>> start >>>>>>>>>>>> invocation fails, while the stop and restart invocations >>>>>>>>>>>> succeed >>>>>>>>>>>> just >>>>>>>>>>>> fine. This makes absolutely no sense because restart is >>>>>>>>>>>> essentially >>>>>>>>>>>> just stop and then start. >>>>>>>>>>>> >>>>>>>>>>>> I think this is actually a bug in systemd. If I reboot the >>>>>>>>>>>> machine, it >>>>>>>>>>>> all works perfectly. If we can't figure out whats going on >>>>>>>>>>>> with >>>>>>>>>>>> systemd, we can always replace this start/stop with >>>>>>>>>>>> starting/stopping >>>>>>>>>>>> the service directly. (pki-tomcatd at pki-tomcat.service). That >>>>>>>>>>>> seems to >>>>>>>>>>>> work all the time with no problems. >>>>>>>>>>>> >>>>>>>>>>>> I suggest we leave this fix (if needed) for a separate patch. >>>>>>>>>>> >>>>>>>>>>> Ok, I will see if I hit this issue again. Btw. is it recommended >>>>>>>>>>> in systemd to >>>>>>>>>>> use target units this way? I thought that only service files are >>>>>>>>>>> meant to be >>>>>>>>>>> used for manipulation with a service. As you said yourself, >>>>>>>>>>> using >>>>>>>>>>> service unit >>>>>>>>>>> file for restart worked every time. >>>>>>>>>>> >>>>>>>>>>> Martin >>>>>>>>>> >>>>>>>>> >>>>>>>>> Now, I tested the rebased patch with VMs with permissive SELinux. >>>>>>>>> IPA server >>>>>>>>> install was OK, as always, but replica installation ended up with >>>>>>>>> error. >>>>>>>>> Although it got much further than before: >>>>>>>>> >>>>>>>>> # ipa-ca-install >>>>>>>>> /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg >>>>>>>>> ... >>>>>>>>> [3/3]: restarting directory server >>>>>>>>> done configuring pkids. >>>>>>>>> Configuring certificate server: Estimated time 3 minutes 30 >>>>>>>>> seconds >>>>>>>>> [1/14]: creating certificate server user >>>>>>>>> [2/14]: configuring certificate server instance >>>>>>>>> [3/14]: disabling nonces >>>>>>>>> [4/14]: importing CA chain to RA certificate database >>>>>>>>> [5/14]: fixing RA database permissions >>>>>>>>> [6/14]: setting up signing cert profile >>>>>>>>> [7/14]: set up CRL publishing >>>>>>>>> [8/14]: set certificate subject base >>>>>>>>> [9/14]: enabling Subject Key Identifier >>>>>>>>> [10/14]: configuring certificate server to start on boot >>>>>>>>> [11/14]: configure certmonger for renewals >>>>>>>>> [12/14]: configure clone certificate renewals >>>>>>>>> [13/14]: configure Server-Cert certificate renewal >>>>>>>>> [14/14]: Configure HTTP to proxy connections >>>>>>>>> done configuring pki-tomcatd. >>>>>>>>> Restarting the directory and certificate servers >>>>>>>>> >>>>>>>>> Your system may be partly configured. >>>>>>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up. >>>>>>>>> >>>>>>>>> It seems like that CA on a replica did not start and so a check >>>>>>>>> for >>>>>>>>> port 8080 >>>>>>>>> failed. Attached logs (pki-ca-logs.tgz) may provide more >>>>>>>>> information >>>>>>>>> to this issue. >>>>>>>>> >>>>>>>> This is the systemd issue I mentioned before. I will submit a >>>>>>>> patch >>>>>>>> which will change the restart mechanism to restart the service >>>>>>>> and not >>>>>>>> the target. >>>>>>>>> >>>>>>> >>>>>>> Patch attached. This patch should be applied on top of the large >>>>>>> patch >>>>>>> (being reviewed). >>>>>>> >>>>>>>>> Now I am also testing upgrade from IPA+dogtag9 to >>>>>>>>> PatchedIPA+dogtag10 >>>>>>>>> (permissive SELinux). Now, the service was upgraded smoothly, CA >>>>>>>>> service >>>>>>>>> could be restarted without failure. However, certificate >>>>>>>>> operations now >>>>>>>>> did not work: >>>>>>>>> >>>>>>>>> # ipactl restart >>>>>>>>> Restarting Directory Service >>>>>>>>> Restarting KDC Service >>>>>>>>> Restarting KPASSWD Service >>>>>>>>> Restarting MEMCACHE Service >>>>>>>>> Restarting HTTP Service >>>>>>>>> Restarting CA Service >>>>>>>>> # ipactl status >>>>>>>>> Directory Service: RUNNING >>>>>>>>> KDC Service: RUNNING >>>>>>>>> KPASSWD Service: RUNNING >>>>>>>>> MEMCACHE Service: RUNNING >>>>>>>>> HTTP Service: RUNNING >>>>>>>>> CA Service: RUNNING >>>>>>>>> # ipa cert-show 1 >>>>>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to >>>>>>>>> communicate with CMS (Internal Server Error) >>>>>>>>> >>>>>>>>> Attaching logs for this case as well (ipa-ca-logs-upgrade.tgz). >>>>>>>>> >>>>>>>> The reason for this is that the old dogtag instances are missing: >>>>>>>> 1. a new parameter in CS.cfg >>>>>>>> 2. a couple of symlinks >>>>>>>> >>>>>>>> I plan to fix this in the dogtag code. Specifically, >>>>>>>> 1. I will modify the dogtag code to provide a default value in >>>>>>>> the case >>>>>>>> where the parameter does not exist. >>>>>>>> 2. I plan to add a function to the dogtag startup scripts to >>>>>>>> check and >>>>>>>> remake any required symlinks. >>>>>>>> >>>>>>>> Both of these changes are in the dogtag code, and do not affect >>>>>>>> this >>>>>>>> patch. As a workaround, to confirm that the upgrade actually >>>>>>>> works, do >>>>>>>> the following manual steps on your upgraded instance: >>>>>>>> >>>>>>>> 1. Add the following parameter to CS.cfg >>>>>>>> pidDir=/var/run >>>>>>>> >>>>>>>> 2. Add the following links; >>>>>>>> cd /var/lib/pki-ca/common/lib >>>>>>>> ln -s /usr/share/java/apache-commons-codec.jar >>>>>>>> apache-commons-codec.jar >>>>>>>> ln -s /usr/share/java/log4j.jar log4j.jar >>>>>>>> >>>>>>>> 3 Restart the dogtag instance >>>>>>>> >>>>>>>>> HTH, >>>>>>>>> Martin >>>>>>>> >>>>>>>> >>>>>> >>>>>> The attached patch conditionalizes the changes, so that IPA supports >>>>>> both Dogtag versions. >>>>>> >>>>>> I didn't put it in platform code for two reasons >>>>>> - One platform (f17) can have either Dogtag version installed >>>>>> - I didn't want to conflict with Timo Aaltonen's "platformizing" >>>>>> work. >>>>>> According to his WIP patches, he plans to move the whole dogtag >>>>>> module >>>>>> into platform code. >>>>>> >>>>>> I used the existence of /usr/bin/pkispawn as a test for Dogtag 10. If >>>>>> you know a better way, please comment. >>>>>> >>>>>> The dogtag_version option is now always added to >>>>>> >>>>>> I've reverted the changes to install/ui/test/data/ipa_init.json, so >>>>>> you'll have to change the ports manually when testing the UI with >>>>>> Dogtag >>>>>> 10. >>>>>> >>>>>> >>>>>> >>>>> >>>>> Attaching a patch with fixed ipa-pki-proxy.conf, as discussed on IRC. >>>>> >>>> >>>> This approach looks good. I did not hit any regression on F17 with >>>> dogtag9, or >>>> clean installs of IPA+dogtag10. I think we are getting close to >>>> pushing this code. >>>> >>>> The only issues I hit so far are for upgraded dogtag9 instances (on >>>> F17): >>>> >>>> 1) The service did not start for me. There were some SELinux AVCs >>>> and even when >>>> I disabled SELinux the instance still did not start (logs attached). >>>> >>>> 2) Uninstall was also not clean, we leave some dogtag installation >>>> states for >>>> upgraded dogtag9 instance: >>>> >>>> # ipa-server-install --uninstall --unattended >>>> Shutting down all IPA services >>>> Removing IPA client configuration >>>> Unconfiguring ntpd >>>> Unconfiguring CA directory server >>>> Unconfiguring web server >>>> Unconfiguring krb5kdc >>>> Unconfiguring kadmin >>>> Unconfiguring directory server >>>> Unconfiguring ipa_memcached >>>> ipa : ERROR Some installation state for pki-cad has not been >>>> restored, see /var/lib/ipa/sysrestore/sysrestore.state >>>> >>>> /var/lib/ipa/sysrestore/sysrestore.state: >>>> [pki-cad] >>>> enabled = False >>> >>> Ade is working on a new build to address the dogtag upgrade issues. >>> >>> The left over state should be removed when we drop the separate instance >>> in the dogtag 9 -> 10 upgrade. I'm not completely sure reading the patch >>> who actually does this part, is this handled by dogtag? >>> >>> rob >>> >> >> The build is available on the developer repo. Just do a yum update. >> >> As for this leftover state, this is an ipa file and such is never >> handled by dogtag. It must be handled somewhere within the ipa code. > > Well, the question is, what handles the dogtag 9 -> dogtag 10 upgrade? > > If IPA isn't involved then it has no chance to remove this state. > > rob > I think the problem is that I treated uninstall as part of the installation code, so it's not looking at the configuration and assuming a dogtag 10-style layout just because the dogtag 10 RPM is installed (but in reality it's dogtag 10 managing the dogtag 9-style layout). I'll look into it soon. -- Petr? From mkosek at redhat.com Thu Sep 6 12:49:55 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 06 Sep 2012 14:49:55 +0200 Subject: [Freeipa-devel] [PATCH] 1053 support 389-ds posix-winsync plugin In-Reply-To: <1346934931.1150.1440.camel@willson.li.ssimo.org> References: <5047950F.6080709@redhat.com> <5047965B.2080908@redhat.com> <50489777.5050700@redhat.com> <1346934931.1150.1440.camel@willson.li.ssimo.org> Message-ID: <50489BF3.8000401@redhat.com> On 09/06/2012 02:35 PM, Simo Sorce wrote: > On Thu, 2012-09-06 at 14:30 +0200, Martin Kosek wrote: >> On 09/05/2012 08:13 PM, Rich Megginson wrote: >>> On 09/05/2012 12:08 PM, Rob Crittenden wrote: >>>> Add support for the 389-ds posix winsync plugin. This plugin will sync the >>>> POSIX attributes from AD. We need to avoid trying to re-add them in our plugin. >>> ack >> >> I did a sanity check, that winsync replication still works, everything looks OK. >> >> Pushed to master, ipa-3-0. > > Is this plugin configurable in IPA ? >>From the commit I can't tell if it is enabled by default or not. > > Simo. > This plugin is configured and enabled by default when IPA server is being installed. You can find LDIF with its configuration there: daemons/ipa-slapi-plugins/ipa-winsync/ipa-winsync-conf.ldif Martin From pviktori at redhat.com Thu Sep 6 12:51:34 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Thu, 06 Sep 2012 14:51:34 +0200 Subject: [Freeipa-devel] [PATCH] 300-301 Fix DNS SOA serial parameters boundaries In-Reply-To: <50473822.20702@redhat.com> References: <50460EB8.2080800@redhat.com> <504725F9.3030509@redhat.com> <504728D9.7020807@redhat.com> <50473822.20702@redhat.com> Message-ID: <50489C56.7090209@redhat.com> On 09/05/2012 01:31 PM, Martin Kosek wrote: > On 09/05/2012 12:26 PM, Petr Viktorin wrote: >> On 09/05/2012 12:14 PM, Petr Viktorin wrote: >>> This works well, but please see some comments below. >>> >>> On 09/04/2012 04:22 PM, Martin Kosek wrote: >>>> To test, simply run the following command: >>>> >>>> ipa dnszone-mod example.com --serial=4294967295 >>>> >>>> This should work well on patched server&client. Web UI should work too >>>> as it >>>> reads the max limit dynamically. >>> >>> Please put this in the test suite. > > Done. > >>> >>>> --- >>>> [PATCH 2/2] Fix DNS SOA serial parameters boundaries: >>>> >>>> Set correct boundaries for DNS SOA serial parameters (see RFC 1035, >>>> 2181). >>>> >>>> >>>> [PATCH 1/2] Transfer long numbers over XMLRPC >>>> >>>> Numeric parameters in ipalib were limited by XMLRPC boundaries for >>>> integer (2^31-1) which is too low for some LDAP attributes like DNS >>>> SOA serial field. >>>> >>>> Transfer numbers which are not in XMLRPC boundary as a string and not >>>> as a number to workaround this limitation. Int parameter had to be >>>> updated to also accept Python's long type as valid int type. >>>> >>>> >>>> freeipa-mkosek-300-transfer-long-numbers-over-xmlrpc.patch >>>> >>>> >>>>> From 8782015a17b130c5ebae8b014a7241810b10dedd Mon Sep 17 00:00:00 2001 >>>> From: Martin Kosek >>>> Date: Tue, 4 Sep 2012 15:49:26 +0200 >>>> Subject: [PATCH 1/2] Transfer long numbers over XMLRPC >>>> >>>> Numeric parameters in ipalib were limited by XMLRPC boundaries for >>>> integer (2^31-1) which is too low for some LDAP attributes like DNS >>>> SOA serial field. >>>> >>>> Transfer numbers which are not in XMLRPC boundary as a string and not >>>> as a number to workaround this limitation. Int parameter had to be >>>> updated to also accept Python's long type as valid int type. >>>> >>>> https://fedorahosted.org/freeipa/ticket/2568 >>>> --- >>>> ipalib/parameters.py | 12 ++++++------ >>>> ipalib/rpc.py | 5 ++++- >>>> 2 files changed, 10 insertions(+), 7 deletions(-) >>>> >>>> diff --git a/ipalib/parameters.py b/ipalib/parameters.py >>>> index >>>> de0d14faf08d1ab79c99e65dab9cc08f406e3a1d..21e30356b2a351bf7a3be7d47d7fabf0130cf6d4 >>>> >>>> 100644 >>>> --- a/ipalib/parameters.py >>>> +++ b/ipalib/parameters.py >>>> @@ -1077,7 +1077,7 @@ class Number(Param): >>>> """ >>>> if type(value) is self.type: >>>> return value >>>> - if type(value) in (unicode, int, float): >>>> + if type(value) in (unicode, int, long, float): >>> >>> >>> PEP8 says that "Object type comparisons should always use isinstance() >>> instead of comparing types directly". >>> It would be nice to change the old code whenever it is touched. It's >>> also in a few more places in the patch. >>> > > I considered doing this when I was developing the patch, but I did not want to > mix type/isinstance in the same code. Now, when I experimented and tried to > replace it in a larger scope, there were unexpected issues. Like bool type > suddenly passing an isinstance(value, (int, long)) test and causing issues later. > > So I skipped this part (as we discussed off-list). > >>>> diff --git a/ipalib/rpc.py b/ipalib/rpc.py >>>> index >>>> d1764e3e30492d5855450398e86689bfcad7aa39..85239ac65903acf447a4d971cce70f819979ce8d >>>> >>>> 100644 >>>> --- a/ipalib/rpc.py >>>> +++ b/ipalib/rpc.py >>>> @@ -94,6 +95,8 @@ def xml_wrap(value): >>>> if type(value) is Decimal: >>>> # transfer Decimal as a string >>>> return unicode(value) >>>> + if isinstance(value, (int, long)) and (value < MININT or value > >>>> MAXINT): >>>> + return unicode(value) >>>> if isinstance(value, DN): >>>> return str(value) >>>> assert type(value) in (unicode, int, float, bool, NoneType) >>> >>> `long` should also be included here. >>> >>> >>> api.Command.user_find(uidnumber=1511000000) >>> {'count': 1, 'truncated': False, 'result': ({...},), 'summary': u'1 user >>> matched'} >>> >>> api.Command.user_find(uidnumber=1511000000 + 0L) >>> Traceback (most recent call last): >>> File "", line 1, in >>> ... >>> File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 102, in >>> xml_wrap >>> assert type(value) in (unicode, int, float, bool, NoneType) >>> AssertionError >>> >>> >>> >>> >> >> One more thing: please update the VERSION file. >> > > I did not want to do this because I just messed with validation, but yeah, I > can do that. > > Fixed patches attached. > Both are good, ACK. -- Petr? From rcritten at redhat.com Thu Sep 6 12:59:20 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 06 Sep 2012 08:59:20 -0400 Subject: [Freeipa-devel] [PATCH] 1053 support 389-ds posix-winsync plugin In-Reply-To: <50489BF3.8000401@redhat.com> References: <5047950F.6080709@redhat.com> <5047965B.2080908@redhat.com> <50489777.5050700@redhat.com> <1346934931.1150.1440.camel@willson.li.ssimo.org> <50489BF3.8000401@redhat.com> Message-ID: <50489E28.8020702@redhat.com> Martin Kosek wrote: > On 09/06/2012 02:35 PM, Simo Sorce wrote: >> On Thu, 2012-09-06 at 14:30 +0200, Martin Kosek wrote: >>> On 09/05/2012 08:13 PM, Rich Megginson wrote: >>>> On 09/05/2012 12:08 PM, Rob Crittenden wrote: >>>>> Add support for the 389-ds posix winsync plugin. This plugin will sync the >>>>> POSIX attributes from AD. We need to avoid trying to re-add them in our plugin. >>>> ack >>> >>> I did a sanity check, that winsync replication still works, everything looks OK. >>> >>> Pushed to master, ipa-3-0. >> >> Is this plugin configurable in IPA ? >> >From the commit I can't tell if it is enabled by default or not. >> >> Simo. >> > > This plugin is configured and enabled by default when IPA server is being > installed. You can find LDIF with its configuration there: > > daemons/ipa-slapi-plugins/ipa-winsync/ipa-winsync-conf.ldif > > Martin I think he means the posix-winsync plugin. It is configured by default in the latest 389-ds build. We do not enable it on upgrades. rob From mkosek at redhat.com Thu Sep 6 13:02:51 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 06 Sep 2012 15:02:51 +0200 Subject: [Freeipa-devel] [PATCH] 300-301 Fix DNS SOA serial parameters boundaries In-Reply-To: <50489C56.7090209@redhat.com> References: <50460EB8.2080800@redhat.com> <504725F9.3030509@redhat.com> <504728D9.7020807@redhat.com> <50473822.20702@redhat.com> <50489C56.7090209@redhat.com> Message-ID: <50489EFB.4000505@redhat.com> On 09/06/2012 02:51 PM, Petr Viktorin wrote: > On 09/05/2012 01:31 PM, Martin Kosek wrote: >> On 09/05/2012 12:26 PM, Petr Viktorin wrote: >>> On 09/05/2012 12:14 PM, Petr Viktorin wrote: >>>> This works well, but please see some comments below. >>>> >>>> On 09/04/2012 04:22 PM, Martin Kosek wrote: >>>>> To test, simply run the following command: >>>>> >>>>> ipa dnszone-mod example.com --serial=4294967295 >>>>> >>>>> This should work well on patched server&client. Web UI should work too >>>>> as it >>>>> reads the max limit dynamically. >>>> >>>> Please put this in the test suite. >> >> Done. >> >>>> >>>>> --- >>>>> [PATCH 2/2] Fix DNS SOA serial parameters boundaries: >>>>> >>>>> Set correct boundaries for DNS SOA serial parameters (see RFC 1035, >>>>> 2181). >>>>> >>>>> >>>>> [PATCH 1/2] Transfer long numbers over XMLRPC >>>>> >>>>> Numeric parameters in ipalib were limited by XMLRPC boundaries for >>>>> integer (2^31-1) which is too low for some LDAP attributes like DNS >>>>> SOA serial field. >>>>> >>>>> Transfer numbers which are not in XMLRPC boundary as a string and not >>>>> as a number to workaround this limitation. Int parameter had to be >>>>> updated to also accept Python's long type as valid int type. >>>>> >>>>> >>>>> freeipa-mkosek-300-transfer-long-numbers-over-xmlrpc.patch >>>>> >>>>> >>>>>> From 8782015a17b130c5ebae8b014a7241810b10dedd Mon Sep 17 00:00:00 2001 >>>>> From: Martin Kosek >>>>> Date: Tue, 4 Sep 2012 15:49:26 +0200 >>>>> Subject: [PATCH 1/2] Transfer long numbers over XMLRPC >>>>> >>>>> Numeric parameters in ipalib were limited by XMLRPC boundaries for >>>>> integer (2^31-1) which is too low for some LDAP attributes like DNS >>>>> SOA serial field. >>>>> >>>>> Transfer numbers which are not in XMLRPC boundary as a string and not >>>>> as a number to workaround this limitation. Int parameter had to be >>>>> updated to also accept Python's long type as valid int type. >>>>> >>>>> https://fedorahosted.org/freeipa/ticket/2568 >>>>> --- >>>>> ipalib/parameters.py | 12 ++++++------ >>>>> ipalib/rpc.py | 5 ++++- >>>>> 2 files changed, 10 insertions(+), 7 deletions(-) >>>>> >>>>> diff --git a/ipalib/parameters.py b/ipalib/parameters.py >>>>> index >>>>> de0d14faf08d1ab79c99e65dab9cc08f406e3a1d..21e30356b2a351bf7a3be7d47d7fabf0130cf6d4 >>>>> >>>>> >>>>> 100644 >>>>> --- a/ipalib/parameters.py >>>>> +++ b/ipalib/parameters.py >>>>> @@ -1077,7 +1077,7 @@ class Number(Param): >>>>> """ >>>>> if type(value) is self.type: >>>>> return value >>>>> - if type(value) in (unicode, int, float): >>>>> + if type(value) in (unicode, int, long, float): >>>> >>>> >>>> PEP8 says that "Object type comparisons should always use isinstance() >>>> instead of comparing types directly". >>>> It would be nice to change the old code whenever it is touched. It's >>>> also in a few more places in the patch. >>>> >> >> I considered doing this when I was developing the patch, but I did not want to >> mix type/isinstance in the same code. Now, when I experimented and tried to >> replace it in a larger scope, there were unexpected issues. Like bool type >> suddenly passing an isinstance(value, (int, long)) test and causing issues >> later. >> >> So I skipped this part (as we discussed off-list). >> >>>>> diff --git a/ipalib/rpc.py b/ipalib/rpc.py >>>>> index >>>>> d1764e3e30492d5855450398e86689bfcad7aa39..85239ac65903acf447a4d971cce70f819979ce8d >>>>> >>>>> >>>>> 100644 >>>>> --- a/ipalib/rpc.py >>>>> +++ b/ipalib/rpc.py >>>>> @@ -94,6 +95,8 @@ def xml_wrap(value): >>>>> if type(value) is Decimal: >>>>> # transfer Decimal as a string >>>>> return unicode(value) >>>>> + if isinstance(value, (int, long)) and (value < MININT or value > >>>>> MAXINT): >>>>> + return unicode(value) >>>>> if isinstance(value, DN): >>>>> return str(value) >>>>> assert type(value) in (unicode, int, float, bool, NoneType) >>>> >>>> `long` should also be included here. >>>> >>>> >>> api.Command.user_find(uidnumber=1511000000) >>>> {'count': 1, 'truncated': False, 'result': ({...},), 'summary': u'1 user >>>> matched'} >>>> >>> api.Command.user_find(uidnumber=1511000000 + 0L) >>>> Traceback (most recent call last): >>>> File "", line 1, in >>>> ... >>>> File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 102, in >>>> xml_wrap >>>> assert type(value) in (unicode, int, float, bool, NoneType) >>>> AssertionError >>>> >>>> >>>> >>>> >>> >>> One more thing: please update the VERSION file. >>> >> >> I did not want to do this because I just messed with validation, but yeah, I >> can do that. >> >> Fixed patches attached. >> > > Both are good, ACK. > Pushed to master, ipa-3-0. Martin From rcritten at redhat.com Thu Sep 6 13:10:55 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 06 Sep 2012 09:10:55 -0400 Subject: [Freeipa-devel] [PATCH] 1041 pull in cachememsize logging In-Reply-To: <50486D59.509@redhat.com> References: <502A522A.1020507@redhat.com> <50479449.2060006@redhat.com> <50486D59.509@redhat.com> Message-ID: <5048A0DF.6050906@redhat.com> Martin Kosek wrote: > On 09/05/2012 08:04 PM, Rob Crittenden wrote: >> Rob Crittenden wrote: >>> 389-ds-base added logging if the entry cache is smaller than the >>> database so users will know they need to tune their DS install. Set this >>> as the minimum for IPA. >>> >>> rob >> >> Rebased patch. >> >> rob >> > > It looks like a changelog entry from your patch 1031-4 slipped in to this > patch. Otherwise it is an obvious patch to ACK. > > Martin > Yeah, I was in rebase hell and messed this up. Fixed and pushed to master and ipa-3-0. rob From simo at redhat.com Thu Sep 6 13:21:05 2012 From: simo at redhat.com (Simo Sorce) Date: Thu, 06 Sep 2012 09:21:05 -0400 Subject: [Freeipa-devel] [PATCH] 1053 support 389-ds posix-winsync plugin In-Reply-To: <50489BF3.8000401@redhat.com> References: <5047950F.6080709@redhat.com> <5047965B.2080908@redhat.com> <50489777.5050700@redhat.com> <1346934931.1150.1440.camel@willson.li.ssimo.org> <50489BF3.8000401@redhat.com> Message-ID: <1346937665.1150.1488.camel@willson.li.ssimo.org> On Thu, 2012-09-06 at 14:49 +0200, Martin Kosek wrote: > On 09/06/2012 02:35 PM, Simo Sorce wrote: > > On Thu, 2012-09-06 at 14:30 +0200, Martin Kosek wrote: > >> On 09/05/2012 08:13 PM, Rich Megginson wrote: > >>> On 09/05/2012 12:08 PM, Rob Crittenden wrote: > >>>> Add support for the 389-ds posix winsync plugin. This plugin will sync the > >>>> POSIX attributes from AD. We need to avoid trying to re-add them in our plugin. > >>> ack > >> > >> I did a sanity check, that winsync replication still works, everything looks OK. > >> > >> Pushed to master, ipa-3-0. > > > > Is this plugin configurable in IPA ? > >>From the commit I can't tell if it is enabled by default or not. > > > > Simo. > > > > This plugin is configured and enabled by default when IPA server is being > installed. You can find LDIF with its configuration there: > > daemons/ipa-slapi-plugins/ipa-winsync/ipa-winsync-conf.ldif Do we want to really set it up by default ? What happen on upgrades ? Simo. -- Simo Sorce * Red Hat, Inc * New York From rmeggins at redhat.com Thu Sep 6 13:32:29 2012 From: rmeggins at redhat.com (Rich Megginson) Date: Thu, 06 Sep 2012 07:32:29 -0600 Subject: [Freeipa-devel] [PATCH] 1053 support 389-ds posix-winsync plugin In-Reply-To: <1346934931.1150.1440.camel@willson.li.ssimo.org> References: <5047950F.6080709@redhat.com> <5047965B.2080908@redhat.com> <50489777.5050700@redhat.com> <1346934931.1150.1440.camel@willson.li.ssimo.org> Message-ID: <5048A5ED.2080901@redhat.com> On 09/06/2012 06:35 AM, Simo Sorce wrote: > On Thu, 2012-09-06 at 14:30 +0200, Martin Kosek wrote: >> On 09/05/2012 08:13 PM, Rich Megginson wrote: >>> On 09/05/2012 12:08 PM, Rob Crittenden wrote: >>>> Add support for the 389-ds posix winsync plugin. This plugin will sync the >>>> POSIX attributes from AD. We need to avoid trying to re-add them in our plugin. >>> ack >> I did a sanity check, that winsync replication still works, everything looks OK. >> >> Pushed to master, ipa-3-0. > Is this plugin configurable in IPA ? Yes. > > From the commit I can't tell if it is enabled by default or not. It is not enabled by default. > > Simo. > From pvoborni at redhat.com Thu Sep 6 14:50:39 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Thu, 06 Sep 2012 16:50:39 +0200 Subject: [Freeipa-devel] [PATCHES] Various fixes for trust and range CLI In-Reply-To: <20120906113947.GD31872@localhost.localdomain> References: <20120906113947.GD31872@localhost.localdomain> Message-ID: <5048B83F.7080309@redhat.com> On 09/06/2012 01:39 PM, Sumit Bose wrote: > Hi, > > this series of patches touches couple of tickets related to the trust > and (id)range CLI. I post them together because some of them depend on > each other. I already rebased them on Martin's "Add range safety check > for range_mod and range_del" patch which I'm currently reviewing. > > bye, > Sumit > > ACK for the UI changes in patch 60. There is a minor issue in ipa_init_objects.json and ipa_init_command.json files: Labels aren't changed so there are still 'Ranges' instead of 'ID ranges'. I don't think it matters because it doesn't affect actual functionality and I will definitely regenerate those files sometime in a future. They are only for developer and related purposes. -- Petr Vobornik From rcritten at redhat.com Thu Sep 6 15:19:20 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 06 Sep 2012 11:19:20 -0400 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <50488DD0.1080507@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> <50479491.1060200@redhat.com> <50488DD0.1080507@redhat.com> Message-ID: <5048BEF8.5000401@redhat.com> Martin Kosek wrote: > On 09/05/2012 08:06 PM, Rob Crittenden wrote: >> Rob Crittenden wrote: >>> Martin Kosek wrote: >>>> On 07/05/2012 08:39 PM, Rob Crittenden wrote: >>>>> Martin Kosek wrote: >>>>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: >>>>>>> Deleting a replica can leave a replication vector (RUV) on the >>>>>>> other servers. >>>>>>> This can confuse things if the replica is re-added, and it also >>>>>>> causes the >>>>>>> server to calculate changes against a server that may no longer exist. >>>>>>> >>>>>>> 389-ds-base provides a new task that self-propogates itself to all >>>>>>> available >>>>>>> replicas to clean this RUV data. >>>>>>> >>>>>>> This patch will create this task at deletion time to hopefully >>>>>>> clean things up. >>>>>>> >>>>>>> It isn't perfect. If any replica is down or unavailable at the time >>>>>>> the >>>>>>> cleanruv task fires, and then comes back up, the old RUV data may be >>>>>>> re-propogated around. >>>>>>> >>>>>>> To make things easier in this case I've added two new commands to >>>>>>> ipa-replica-manage. The first lists the replication ids of all the >>>>>>> servers we >>>>>>> have a RUV for. Using this you can call clean_ruv with the >>>>>>> replication id of a >>>>>>> server that no longer exists to try the cleanallruv step again. >>>>>>> >>>>>>> This is quite dangerous though. If you run cleanruv against a >>>>>>> replica id that >>>>>>> does exist it can cause a loss of data. I believe I've put in >>>>>>> enough scary >>>>>>> warnings about this. >>>>>>> >>>>>>> rob >>>>>>> >>>>>> >>>>>> Good work there, this should make cleaning RUVs much easier than >>>>>> with the >>>>>> previous version. >>>>>> >>>>>> This is what I found during review: >>>>>> >>>>>> 1) list_ruv and clean_ruv command help in man is quite lost. I think >>>>>> it would >>>>>> help if we for example have all info for commands indented. This way >>>>>> user could >>>>>> simply over-look the new commands in the man page. >>>>>> >>>>>> >>>>>> 2) I would rename new commands to clean-ruv and list-ruv to make them >>>>>> consistent with the rest of the commands (re-initialize, force-sync). >>>>>> >>>>>> >>>>>> 3) It would be nice to be able to run clean_ruv command in an >>>>>> unattended way >>>>>> (for better testing), i.e. respect --force option as we already do for >>>>>> ipa-replica-manage del. This fix would aid test automation in the >>>>>> future. >>>>>> >>>>>> >>>>>> 4) (minor) The new question (and the del too) does not react too >>>>>> well for >>>>>> CTRL+D: >>>>>> >>>>>> # ipa-replica-manage clean_ruv 3 --force >>>>>> Clean the Replication Update Vector for >>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>> >>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>> longer replicate so it may miss updates while the process >>>>>> is running. It would need to be re-initialized to maintain >>>>>> consistency. Be very careful. >>>>>> Continue to clean? [no]: unexpected error: >>>>>> >>>>>> >>>>>> 5) Help for clean_ruv command without a required parameter is quite >>>>>> confusing >>>>>> as it reports that command is wrong and not the parameter: >>>>>> >>>>>> # ipa-replica-manage clean_ruv >>>>>> Usage: ipa-replica-manage [options] >>>>>> >>>>>> ipa-replica-manage: error: must provide a command [clean_ruv | >>>>>> force-sync | >>>>>> disconnect | connect | del | re-initialize | list | list_ruv] >>>>>> >>>>>> It seems you just forgot to specify the error message in the command >>>>>> definition >>>>>> >>>>>> >>>>>> 6) When the remote replica is down, the clean_ruv command fails with an >>>>>> unexpected error: >>>>>> >>>>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 >>>>>> Clean the Replication Update Vector for >>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>> >>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>> longer replicate so it may miss updates while the process >>>>>> is running. It would need to be re-initialized to maintain >>>>>> consistency. Be very careful. >>>>>> Continue to clean? [no]: y >>>>>> unexpected error: {'desc': 'Operations error'} >>>>>> >>>>>> >>>>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: >>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>> cleanAllRUV_task: failed >>>>>> to connect to repl agreement connection >>>>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, >>>>>> >>>>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>> tree,cn=config), error 105 >>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>> cleanAllRUV_task: replica >>>>>> (cn=meTovm-055.idm.lab. >>>>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>> >>>>>> >>>>>> >>>>>> tree, cn=config) has not been cleaned. You will need to rerun the >>>>>> CLEANALLRUV task on this replica. >>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>> cleanAllRUV_task: Task >>>>>> failed (1) >>>>>> >>>>>> In this case I think we should inform user that the command failed, >>>>>> possibly >>>>>> because of disconnected replicas and that they could enable the >>>>>> replicas and >>>>>> try again. >>>>>> >>>>>> >>>>>> 7) (minor) "pass" is now redundant in replication.py: >>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>> + # We can't make the server we're removing read-only but >>>>>> + # this isn't a show-stopper >>>>>> + root_logger.debug("No permission to switch replica to >>>>>> read-only, >>>>>> continuing anyway") >>>>>> + pass >>>>>> >>>>> >>>>> I think this addresses everything. >>>>> >>>>> rob >>>> >>>> Thanks, almost there! I just found one more issue which needs to be fixed >>>> before we push: >>>> >>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>> Directory Manager password: >>>> >>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing >>>> removal >>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': "Can't >>>> contact LDAP server"} >>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>> >>>> There were issues removing a connection: %d format: a number is >>>> required, not str >>>> >>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': "Can't >>>> contact LDAP server"} >>>> >>>> This is a traceback I retrieved: >>>> Traceback (most recent call last): >>>> File "/sbin/ipa-replica-manage", line 425, in del_master >>>> del_link(realm, r, hostname, options.dirman_passwd, force=True) >>>> File "/sbin/ipa-replica-manage", line 271, in del_link >>>> repl1.cleanallruv(replica_id) >>>> File >>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>> line 1094, in cleanallruv >>>> root_logger.debug("Creating CLEANALLRUV task for replica id %d" % >>>> replicaId) >>>> >>>> >>>> The problem here is that you don't convert replica_id to int in this >>>> part: >>>> + replica_id = None >>>> + if repl2: >>>> + replica_id = repl2._get_replica_id(repl2.conn, None) >>>> + else: >>>> + servers = get_ruv(realm, replica1, dirman_passwd) >>>> + for (netloc, rid) in servers: >>>> + if netloc.startswith(replica2): >>>> + replica_id = rid >>>> + break >>>> >>>> Martin >>>> >>> >>> Updated patch using new mechanism in 389-ds-base. This should more >>> thoroughly clean out RUV data when a replica is being deleted, and >>> provide for a way to delete RUV data afterwards too if necessary. >>> >>> rob >> >> Rebased patch >> >> rob >> > > 0) As I wrote in a review for your patch 1041, changelog entry slipped elsewhere. > > 1) The following KeyboardInterrupt except class looks suspicious. I know why > you have it there, but since it is generally a bad thing to do, some comment > why it is needed would be useful. > > @@ -256,6 +263,17 @@ def del_link(realm, replica1, replica2, dirman_passwd, > force=False): > repl1.delete_agreement(replica2) > repl1.delete_referral(replica2) > > + if type1 == replication.IPA_REPLICA: > + if repl2: > + ruv = repl2._get_replica_id(repl2.conn, None) > + else: > + ruv = get_ruv_by_host(realm, replica1, replica2, dirman_passwd) > + > + try: > + repl1.cleanallruv(ruv) > + except KeyboardInterrupt: > + pass > + > > Maybe you just wanted to do some cleanup and then "raise" again? No, it is there because it is safe to break out of it. The task will continue to run. I added some verbiage. > > 2) This is related to 1), but when some replica is down, "ipa-replica-manage > del" may wait indefinitely when some remote replica is down, right? > > # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com > Deleting a master is irreversible. > To reconnect to the remote master you will need to prepare a new replica file > and re-install. > Continue to delete? [no]: y > ipa: INFO: Setting agreement > cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping > tree,cn=config schedule to 2358-2359 0 to force synch > ipa: INFO: Deleting schedule 2358-2359 0 from agreement > cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping > tree,cn=config > ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica acquired > successfully: Incremental update succeeded: start: 0: end: 0 > Background task created to clean replication data > > ... after about a minute I hit CTRL+C > > ^CDeleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to > 'vm-055.idm.lab.bos.redhat.com' > Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: NS record does not > contain 'vm-055.idm.lab.bos.redhat.com.' > You may need to manually remove them from the tree > > I think it would be better to inform user that some remote replica is down or > at least that we are waiting for the task to complete. Something like that: > > # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com > ... > Background task created to clean replication data > Replication data clean up may take very long time if some replica is unreachable > Hit CTRL+C to interrupt the wait > ^C Clean up wait interrupted > .... > [continue with del] Yup, did this in #1. > > 3) (minor) When there is a cleanruv task running and you run > "ipa-replica-manage del", there is a unexpected error message with duplicate > task object in LDAP: > > # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com --force > Unable to connect to replica vm-072.idm.lab.bos.redhat.com, forcing removal > FAIL > Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't > contact LDAP server"} > Forcing removal on 'vm-086.idm.lab.bos.redhat.com' > > There were issues removing a connection: This entry already exists <<<<<<<<< > > Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't > contact LDAP server"} > Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record does not > contain 'vm-072.idm.lab.bos.redhat.com.' > You may need to manually remove them from the tree > > > I think it should be enough to just catch for "entry already exists" in > cleanallruv function, and in such case print a relevant error message bail out. > Thus, self.conn.checkTask(dn, dowait=True) would not be called too. Good catch, fixed. > > > 4) (minor): In make_readonly function, there is a redundant "pass" statement: > > + def make_readonly(self): > + """ > + Make the current replication agreement read-only. > + """ > + dn = DN(('cn', 'userRoot'), ('cn', 'ldbm database'), > + ('cn', 'plugins'), ('cn', 'config')) > + > + mod = [(ldap.MOD_REPLACE, 'nsslapd-readonly', 'on')] > + try: > + self.conn.modify_s(dn, mod) > + except ldap.INSUFFICIENT_ACCESS: > + # We can't make the server we're removing read-only but > + # this isn't a show-stopper > + root_logger.debug("No permission to switch replica to read-only, > continuing anyway") > + pass <<<<<<<<<<<<<<< Yeah, this is one of my common mistakes. I put in a pass initially, then add logging in front of it and forget to delete the pass. Its gone now. > > > 5) In clean_ruv, I think allowing a --force option to bypass the user_input > would be helpful (at least for test automation): > > + if not ipautil.user_input("Continue to clean?", False): > + sys.exit("Aborted") Yup, added. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-rcrit-1031-5-cleanruv.patch Type: text/x-diff Size: 14216 bytes Desc: not available URL: From mkosek at redhat.com Thu Sep 6 15:22:21 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 06 Sep 2012 17:22:21 +0200 Subject: [Freeipa-devel] [PATCH] 1050 prevent replica orphans In-Reply-To: <5040F722.4060208@redhat.com> References: <5040F6E2.7060109@redhat.com> <5040F722.4060208@redhat.com> Message-ID: <5048BFAD.1070807@redhat.com> On 08/31/2012 07:40 PM, Rob Crittenden wrote: > Rob Crittenden wrote: >> It was possible use ipa-replica-manage connect/disconnect/del to end up >> orphaning or or more IPA masters. This is an attempt to catch and >> prevent that case. >> >> I tested with this topology, trying to delete B. >> >> A <-> B <-> C >> >> I got here by creating B and C from A, connecting B to C then deleting >> the link from A to B, so it went from A -> B and A -> C to the above. >> >> What I do is look up the servers that the delete candidate host has >> connections to and see if we're the last link. >> >> I added an escape clause if there are only two masters. >> >> rob > > Oh, this relies on my cleanruv patch 1031. > > rob > 1) When I run ipa-replica-manage del --force to an already uninstalled host, the new code will prevent me the deletation because it cannot connect to it. It also crashes with UnboundLocalError: # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing removal Traceback (most recent call last): File "/sbin/ipa-replica-manage", line 708, in main() File "/sbin/ipa-replica-manage", line 677, in main del_master(realm, args[1], options) File "/sbin/ipa-replica-manage", line 476, in del_master sys.exit("Failed read master data from '%s': %s" % (delrepl.hostname, str(e))) UnboundLocalError: local variable 'delrepl' referenced before assignment I also hit this error when removing a winsync replica. 2) As I wrote before, I think having --force option override the user inquiries would benefit test automation: + if not ipautil.user_input("Continue to delete?", False): + sys.exit("Aborted") 3) I don't think this code won't cover this topology: A - B - C - D - E It would allow you deleting a replica C even though it would separate A-B and D-E. Though we may not want to cover this situation now, what you got is definitely helping. Martin From rcritten at redhat.com Thu Sep 6 15:42:07 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 06 Sep 2012 11:42:07 -0400 Subject: [Freeipa-devel] [PATCH] 1046 add e-mail by default In-Reply-To: <50489AE2.7020709@redhat.com> References: <5037BFEB.8050602@redhat.com> <50489AE2.7020709@redhat.com> Message-ID: <5048C44F.3030106@redhat.com> Martin Kosek wrote: > On 08/24/2012 07:54 PM, Rob Crittenden wrote: >> We weren't automatically creating the mail attribute despite having the default >> e-mail domain. This patch will add it to all new users. >> >> To disable creating this set the default e-mail domain to empty in ipa config. >> >> rob >> > > 1) Patch needs a rebase > > 2) There are 2 test cases where new default mail attribute was not added: > > ====================================================================== > FAIL: test_user[34]: user_find: Search for "tuser2" with manager "tuser1" > ---------------------------------------------------------------------- > ... > extra keys = ['mail'] > ... > > ====================================================================== > FAIL: test_user[75]: user_add: Create 2nd admin user "admin2" > ---------------------------------------------------------------------- > ... > extra keys = ['mail'] > ... > > 3) Some code could be simplified: > > This: > + if 'ipadefaultemaildomain' in config: > + defaultdomain = config['ipadefaultemaildomain'][0] > + else: > + defaultdomain = None > > To this: > defaultdomain = config.get('ipadefaultemaildomain', [None])[0] > > > This: > if m.find('@') == -1 ... > > To this: > if '@' not in m ... > > IMHO, it is more readable than the find method. > > 3) When default e-mail domain is removed from config, users cannot be added any > more when e-mail is not explicitly specified: > > # ipa config-mod --emaildomain= > Maximum username length: 32 > Home directory base: /home > Default shell: /bin/sh > Default users group: ipausers > Search time limit: 2 > Search size limit: 100 > User search fields: uid,givenname,sn,telephonenumber,ou,title > Group search fields: cn,description > Enable migration mode: FALSE > Certificate Subject base: O=IDM.LAB.BOS.REDHAT.COM > Password Expiration Notification (days): 4 > Password plugin features: AllowNThash > SELinux user map order: > guest_u:s0$xguest_u:s0$user_u:s0-s0:c0.c1023$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023 > Default SELinux user: guest_u:s0 > PAC type: MS-PAC > > # ipa user-add --first=Foo --last=Bar fbar > ipa: ERROR: invalid 'email': invalid e-mail format: fbar > > Martin > Rebased, issues addressed. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-rcrit-1046-2-mail.patch Type: text/x-diff Size: 21313 bytes Desc: not available URL: From rcritten at redhat.com Thu Sep 6 15:42:34 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 06 Sep 2012 11:42:34 -0400 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <5048BEF8.5000401@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> <50479491.1060200@redhat.com> <50488DD0.1080507@redhat.com> <5048BEF8.5000401@redhat.com> Message-ID: <5048C46A.8020207@redhat.com> Rob Crittenden wrote: > Martin Kosek wrote: >> On 09/05/2012 08:06 PM, Rob Crittenden wrote: >>> Rob Crittenden wrote: >>>> Martin Kosek wrote: >>>>> On 07/05/2012 08:39 PM, Rob Crittenden wrote: >>>>>> Martin Kosek wrote: >>>>>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: >>>>>>>> Deleting a replica can leave a replication vector (RUV) on the >>>>>>>> other servers. >>>>>>>> This can confuse things if the replica is re-added, and it also >>>>>>>> causes the >>>>>>>> server to calculate changes against a server that may no longer >>>>>>>> exist. >>>>>>>> >>>>>>>> 389-ds-base provides a new task that self-propogates itself to all >>>>>>>> available >>>>>>>> replicas to clean this RUV data. >>>>>>>> >>>>>>>> This patch will create this task at deletion time to hopefully >>>>>>>> clean things up. >>>>>>>> >>>>>>>> It isn't perfect. If any replica is down or unavailable at the time >>>>>>>> the >>>>>>>> cleanruv task fires, and then comes back up, the old RUV data >>>>>>>> may be >>>>>>>> re-propogated around. >>>>>>>> >>>>>>>> To make things easier in this case I've added two new commands to >>>>>>>> ipa-replica-manage. The first lists the replication ids of all the >>>>>>>> servers we >>>>>>>> have a RUV for. Using this you can call clean_ruv with the >>>>>>>> replication id of a >>>>>>>> server that no longer exists to try the cleanallruv step again. >>>>>>>> >>>>>>>> This is quite dangerous though. If you run cleanruv against a >>>>>>>> replica id that >>>>>>>> does exist it can cause a loss of data. I believe I've put in >>>>>>>> enough scary >>>>>>>> warnings about this. >>>>>>>> >>>>>>>> rob >>>>>>>> >>>>>>> >>>>>>> Good work there, this should make cleaning RUVs much easier than >>>>>>> with the >>>>>>> previous version. >>>>>>> >>>>>>> This is what I found during review: >>>>>>> >>>>>>> 1) list_ruv and clean_ruv command help in man is quite lost. I think >>>>>>> it would >>>>>>> help if we for example have all info for commands indented. This way >>>>>>> user could >>>>>>> simply over-look the new commands in the man page. >>>>>>> >>>>>>> >>>>>>> 2) I would rename new commands to clean-ruv and list-ruv to make >>>>>>> them >>>>>>> consistent with the rest of the commands (re-initialize, >>>>>>> force-sync). >>>>>>> >>>>>>> >>>>>>> 3) It would be nice to be able to run clean_ruv command in an >>>>>>> unattended way >>>>>>> (for better testing), i.e. respect --force option as we already >>>>>>> do for >>>>>>> ipa-replica-manage del. This fix would aid test automation in the >>>>>>> future. >>>>>>> >>>>>>> >>>>>>> 4) (minor) The new question (and the del too) does not react too >>>>>>> well for >>>>>>> CTRL+D: >>>>>>> >>>>>>> # ipa-replica-manage clean_ruv 3 --force >>>>>>> Clean the Replication Update Vector for >>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>> >>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>> longer replicate so it may miss updates while the process >>>>>>> is running. It would need to be re-initialized to maintain >>>>>>> consistency. Be very careful. >>>>>>> Continue to clean? [no]: unexpected error: >>>>>>> >>>>>>> >>>>>>> 5) Help for clean_ruv command without a required parameter is quite >>>>>>> confusing >>>>>>> as it reports that command is wrong and not the parameter: >>>>>>> >>>>>>> # ipa-replica-manage clean_ruv >>>>>>> Usage: ipa-replica-manage [options] >>>>>>> >>>>>>> ipa-replica-manage: error: must provide a command [clean_ruv | >>>>>>> force-sync | >>>>>>> disconnect | connect | del | re-initialize | list | list_ruv] >>>>>>> >>>>>>> It seems you just forgot to specify the error message in the command >>>>>>> definition >>>>>>> >>>>>>> >>>>>>> 6) When the remote replica is down, the clean_ruv command fails >>>>>>> with an >>>>>>> unexpected error: >>>>>>> >>>>>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 >>>>>>> Clean the Replication Update Vector for >>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>> >>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>> longer replicate so it may miss updates while the process >>>>>>> is running. It would need to be re-initialized to maintain >>>>>>> consistency. Be very careful. >>>>>>> Continue to clean? [no]: y >>>>>>> unexpected error: {'desc': 'Operations error'} >>>>>>> >>>>>>> >>>>>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: >>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>> cleanAllRUV_task: failed >>>>>>> to connect to repl agreement connection >>>>>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, >>>>>>> >>>>>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>> >>>>>>> tree,cn=config), error 105 >>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>> cleanAllRUV_task: replica >>>>>>> (cn=meTovm-055.idm.lab. >>>>>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> tree, cn=config) has not been cleaned. You will need to rerun the >>>>>>> CLEANALLRUV task on this replica. >>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>> cleanAllRUV_task: Task >>>>>>> failed (1) >>>>>>> >>>>>>> In this case I think we should inform user that the command failed, >>>>>>> possibly >>>>>>> because of disconnected replicas and that they could enable the >>>>>>> replicas and >>>>>>> try again. >>>>>>> >>>>>>> >>>>>>> 7) (minor) "pass" is now redundant in replication.py: >>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>> + # We can't make the server we're removing read-only but >>>>>>> + # this isn't a show-stopper >>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>> read-only, >>>>>>> continuing anyway") >>>>>>> + pass >>>>>>> >>>>>> >>>>>> I think this addresses everything. >>>>>> >>>>>> rob >>>>> >>>>> Thanks, almost there! I just found one more issue which needs to be >>>>> fixed >>>>> before we push: >>>>> >>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>>> Directory Manager password: >>>>> >>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing >>>>> removal >>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>> "Can't >>>>> contact LDAP server"} >>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>> >>>>> There were issues removing a connection: %d format: a number is >>>>> required, not str >>>>> >>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>> "Can't >>>>> contact LDAP server"} >>>>> >>>>> This is a traceback I retrieved: >>>>> Traceback (most recent call last): >>>>> File "/sbin/ipa-replica-manage", line 425, in del_master >>>>> del_link(realm, r, hostname, options.dirman_passwd, force=True) >>>>> File "/sbin/ipa-replica-manage", line 271, in del_link >>>>> repl1.cleanallruv(replica_id) >>>>> File >>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>> line 1094, in cleanallruv >>>>> root_logger.debug("Creating CLEANALLRUV task for replica id >>>>> %d" % >>>>> replicaId) >>>>> >>>>> >>>>> The problem here is that you don't convert replica_id to int in this >>>>> part: >>>>> + replica_id = None >>>>> + if repl2: >>>>> + replica_id = repl2._get_replica_id(repl2.conn, None) >>>>> + else: >>>>> + servers = get_ruv(realm, replica1, dirman_passwd) >>>>> + for (netloc, rid) in servers: >>>>> + if netloc.startswith(replica2): >>>>> + replica_id = rid >>>>> + break >>>>> >>>>> Martin >>>>> >>>> >>>> Updated patch using new mechanism in 389-ds-base. This should more >>>> thoroughly clean out RUV data when a replica is being deleted, and >>>> provide for a way to delete RUV data afterwards too if necessary. >>>> >>>> rob >>> >>> Rebased patch >>> >>> rob >>> >> >> 0) As I wrote in a review for your patch 1041, changelog entry slipped >> elsewhere. >> >> 1) The following KeyboardInterrupt except class looks suspicious. I >> know why >> you have it there, but since it is generally a bad thing to do, some >> comment >> why it is needed would be useful. >> >> @@ -256,6 +263,17 @@ def del_link(realm, replica1, replica2, >> dirman_passwd, >> force=False): >> repl1.delete_agreement(replica2) >> repl1.delete_referral(replica2) >> >> + if type1 == replication.IPA_REPLICA: >> + if repl2: >> + ruv = repl2._get_replica_id(repl2.conn, None) >> + else: >> + ruv = get_ruv_by_host(realm, replica1, replica2, >> dirman_passwd) >> + >> + try: >> + repl1.cleanallruv(ruv) >> + except KeyboardInterrupt: >> + pass >> + >> >> Maybe you just wanted to do some cleanup and then "raise" again? > > No, it is there because it is safe to break out of it. The task will > continue to run. I added some verbiage. > >> >> 2) This is related to 1), but when some replica is down, >> "ipa-replica-manage >> del" may wait indefinitely when some remote replica is down, right? >> >> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >> Deleting a master is irreversible. >> To reconnect to the remote master you will need to prepare a new >> replica file >> and re-install. >> Continue to delete? [no]: y >> ipa: INFO: Setting agreement >> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >> >> tree,cn=config schedule to 2358-2359 0 to force synch >> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >> >> tree,cn=config >> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica >> acquired >> successfully: Incremental update succeeded: start: 0: end: 0 >> Background task created to clean replication data >> >> ... after about a minute I hit CTRL+C >> >> ^CDeleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to >> 'vm-055.idm.lab.bos.redhat.com' >> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: NS record >> does not >> contain 'vm-055.idm.lab.bos.redhat.com.' >> You may need to manually remove them from the tree >> >> I think it would be better to inform user that some remote replica is >> down or >> at least that we are waiting for the task to complete. Something like >> that: >> >> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >> ... >> Background task created to clean replication data >> Replication data clean up may take very long time if some replica is >> unreachable >> Hit CTRL+C to interrupt the wait >> ^C Clean up wait interrupted >> .... >> [continue with del] > > Yup, did this in #1. > >> >> 3) (minor) When there is a cleanruv task running and you run >> "ipa-replica-manage del", there is a unexpected error message with >> duplicate >> task object in LDAP: >> >> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com --force >> Unable to connect to replica vm-072.idm.lab.bos.redhat.com, forcing >> removal >> FAIL >> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >> contact LDAP server"} >> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >> >> There were issues removing a connection: This entry already exists >> <<<<<<<<< >> >> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >> contact LDAP server"} >> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record >> does not >> contain 'vm-072.idm.lab.bos.redhat.com.' >> You may need to manually remove them from the tree >> >> >> I think it should be enough to just catch for "entry already exists" in >> cleanallruv function, and in such case print a relevant error message >> bail out. >> Thus, self.conn.checkTask(dn, dowait=True) would not be called too. > > Good catch, fixed. > >> >> >> 4) (minor): In make_readonly function, there is a redundant "pass" >> statement: >> >> + def make_readonly(self): >> + """ >> + Make the current replication agreement read-only. >> + """ >> + dn = DN(('cn', 'userRoot'), ('cn', 'ldbm database'), >> + ('cn', 'plugins'), ('cn', 'config')) >> + >> + mod = [(ldap.MOD_REPLACE, 'nsslapd-readonly', 'on')] >> + try: >> + self.conn.modify_s(dn, mod) >> + except ldap.INSUFFICIENT_ACCESS: >> + # We can't make the server we're removing read-only but >> + # this isn't a show-stopper >> + root_logger.debug("No permission to switch replica to >> read-only, >> continuing anyway") >> + pass <<<<<<<<<<<<<<< > > Yeah, this is one of my common mistakes. I put in a pass initially, then > add logging in front of it and forget to delete the pass. Its gone now. > >> >> >> 5) In clean_ruv, I think allowing a --force option to bypass the >> user_input >> would be helpful (at least for test automation): >> >> + if not ipautil.user_input("Continue to clean?", False): >> + sys.exit("Aborted") > > Yup, added. > > rob Slightly revised patch. I still had a window open with one unsaved change. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-rcrit-1031-6-cleanruv.patch Type: text/x-diff Size: 14660 bytes Desc: not available URL: From pvoborni at redhat.com Thu Sep 6 15:43:26 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Thu, 06 Sep 2012 17:43:26 +0200 Subject: [Freeipa-devel] [PATCH] 204 Update of confirmation of actions In-Reply-To: <50486BD7.2050200@redhat.com> References: <5044D541.2000904@redhat.com> <5047A869.4040906@redhat.com> <50486BD7.2050200@redhat.com> Message-ID: <5048C49E.7060301@redhat.com> On 09/06/2012 11:24 AM, Petr Vobornik wrote: > On 09/05/2012 09:30 PM, Endi Sukma Dewata wrote: >> On 9/3/2012 11:05 AM, Petr Vobornik wrote: >>> This patch is changing confirmation of actions according to ticket >>> #3035, see the ticket description. >>> >>> It does following changes: >>> * Confirmation of update action was removed. >>> * Action lists resets to first action (which is usually a NOP: '-- >>> select action --') on change of displayed entry. >>> * New confirmation dialog was implemented. It is used for action >>> confirmation. It is used in IPA.action to replace the call of >>> window.confirm(message). The old call is a modal window which blocks all >>> JS functionality and has different style than other dialogs in Web UI. >>> The new one has same design and doesn't block background operations. >>> >>> https://fedorahosted.org/freeipa/ticket/3035 >> >> Some issues: >> >> 1. None of the confirmation dialogs have a default button. If you hit >> Enter it will do nothing (except #2 below). All buttons are greyed out >> until you hover with mouse or focus with Tab. Is this intentional? I >> think usually a confirmation dialog would have a default button. >> >> 2. In the Users search facet the confirmation dialog doesn't show >> default button either, but if you hit Enter it will execute the >> operation. This is inconsistent with #1. >> > > Root of the problem is that old custom dialogs which serves for > confirmation (batch delete, various certificate and unprovisioning > dialogs) or other dialogs (password change, adder dialogs) don't have > confirmation by enter. Only the new dialog has confirmation by Enter. At > the moment this new dialog is used only for mass enable/disable. > > We might: > a) remove confirmation confirmation by pressing Enter key from the new > dialog to be consistent. IMO wrong. > b) add this confirmation too all dialogs > c) change old confirm dialogs to inherit from IPA.confirm_dialog to > receive this functionality > > IMO, for most dialogs c) would be better but for some (highly > customized) b). It should be probably done in other patch, maybe it the > context of this ticket. > > There is also: https://fedorahosted.org/freeipa/ticket/2910 ([Web UI] > Use "Enter" to confirm add dialogs) > Pushed to master, ipa 3-0 I agreed with Endi over IRC that this patch will be pushed to make to RC1. Ticket will remain opened and moved to RC2 where combination of (b) and (c) will be addressed. -- Petr Vobornik From jcholast at redhat.com Thu Sep 6 15:47:44 2012 From: jcholast at redhat.com (Jan Cholasta) Date: Thu, 06 Sep 2012 17:47:44 +0200 Subject: [Freeipa-devel] [PATCH] 83 Use OpenSSH-style public keys as the preferred format of SSH public keys In-Reply-To: <5047BCBE.1090304@redhat.com> References: <50463229.5030305@redhat.com> <5047BCBE.1090304@redhat.com> Message-ID: <5048C5A0.5090204@redhat.com> Dne 5.9.2012 22:57, Rob Crittenden napsal(a): > Jan Cholasta wrote: >> Hi, >> >> this patch changes the format of the sshpubkey parameter to the format >> used by OpenSSH (see sshd(8)). >> >> Public keys in the old format (raw RFC 4253 blob) are automatically >> converted to OpenSSH-style public keys. OpenSSH-style public keys are >> now stored in LDAP. >> >> Changed sshpubkeyfp to be an output parameter, as that is what it >> actually is. >> >> Allow parameter normalizers to be used on values of any type, not just >> unicode, so that public key blobs (which are str) can be normalized to >> OpenSSH-style public keys. >> >> Note that you need a SSSD build including >> >> >> (SSSD 1.9.0beta7-to-be) in order to make OpenSSH integration actually >> work with OpenSSH-style public keys. >> >> >> >> >> Honza > > NACK. > > I think a bunch of tests are needed for this. > > Because you abstracted out the pubkey class it should be straightforward > to add a bunch of class-based unit tests on it. > > There are also no user or host-based tests, either for adding or > managing keys. Tests added. > > I tested backwards compatibility with 2.2 and the initial tests are mixed. > > I installed 2.2 and created a 3.0 clone from it, including your patch. Do people actually do that in real deployments? > > I added a user in 3.0 with a key and it added ok, but on the 2.2 side it > returns the entire base64 encoded blob of key type, key and comment, > which I presume is unusable. At least things don't blow up. The format of ipasshpubkey in LDAP has changed, so there's not much I can do about this. > > The reverse works fine. An old-style key added to 2.2 appears to work > fine in 3.0, we just lack a comment. > > On the 2.2 server: > > $ ipa user-show tuser1 --all | grep -i ssh > Base-64 encoded SSH public key: > c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDNUQyRTI2dHU5YXM2cHhlUVlSdUgzelYyUDUzMjFpR1U5aC9XNElpd0tGSGlOc2p5cXFyemhCUFB3am83dGlYRDlHbUo1M25KS21OTGd0K01XUnFTZEx2R0V3NjM3SkVTWEpGL0VWeUxvZEFWRGltdXFRVkNLWjBRcm1kYjErRUg1VGRrd3ByOExyd0g1a0RzMEVpcGc2c0xoRUZ5NzMvaXNjRkJqcmk0NGxSU1BZNXFHTWFLOVE0cjY1WFEyaytlZ1RDQnBNZnc0b0J6Mzh0ZHVEVVE2bW9XNFhQSnhZeWJ3MGFDMnRUK2RBOU42WndFSFZXREUzdzg0bHRHa0JRZFRaKzViRnBFdlladm9PbkZXdDlNZFIzYVd6UklnY1o5VDlySDFFT2Z3eE5zWVRCLzRjTmg3dS9adGxnMVV0Z1VteWN3TkpMTUYrMTNzNTl2OFFpSFogcmNyaXRAZWRzZWwuZ3JleW9hay5jb20= > > $ python > Python 2.7.3 (default, Jul 24 2012, 10:05:38) > [GCC 4.7.0 20120507 (Red Hat 4.7.0-5)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import base64 > >>> s = > 'c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDNUQyRTI2dHU5YXM2cHhlUVlSdUgzelYyUDUzMjFpR1U5aC9XNElpd0tGSGlOc2p5cXFyemhCUFB3am83dGlYRDlHbUo1M25KS21OTGd0K01XUnFTZEx2R0V3NjM3SkVTWEpGL0VWeUxvZEFWRGltdXFRVkNLWjBRcm1kYjErRUg1VGRrd3ByOExyd0g1a0RzMEVpcGc2c0xoRUZ5NzMvaXNjRkJqcmk0NGxSU1BZNXFHTWFLOVE0cjY1WFEyaytlZ1RDQnBNZnc0b0J6Mzh0ZHVEVVE2bW9XNFhQSnhZeWJ3MGFDMnRUK2RBOU42WndFSFZXREUzdzg0bHRHa0JRZFRaKzViRnBFdlladm9PbkZXdDlNZFIzYVd6UklnY1o5VDlySDFFT2Z3eE5zWVRCLzRjTmg3dS9adGxnMVV0Z1VteWN3TkpMTUYrMTNzNTl2OFFpSFogcmNyaXRAZWRzZWwuZ3JleW9hay5jb20=' > > >>> base64.b64decode(s) > 'ssh-rsa > AAAAB3NzaC1yc2EAAAADAQABAAABAQC5D2E26tu9as6pxeQYRuH3zV2P5321iGU9h/W4IiwKFHiNsjyqqrzhBPPwjo7tiXD9GmJ53nJKmNLgt+MWRqSdLvGEw637JESXJF/EVyLodAVDimuqQVCKZ0Qrmdb1+EH5Tdkwpr8LrwH5kDs0Eipg6sLhEFy73/iscFBjri44lRSPY5qGMaK9Q4r65XQ2k+egTCBpMfw4oBz38tduDUQ6moW4XPJxYybw0aC2tT+dA9N6ZwEHVWDE3w84ltGkBQdTZ+5bFpEvYZvoOnFWt9MdR3aWzRIgcZ9T9rH1EOfwxNsYTB/4cNh7u/Ztlg1UtgUmycwNJLMF+13s59v8QiHZ > rcrit at edsel.greyoak.com' > > Now show an old style key: > > $ ipa user-show tuser2 --all | grep -i ssh > Base-64 encoded SSH public key: > AAAAB3NzaC1yc2EAAAADAQABAAABAQCbRLyizFGyfucNRnHpWdUG8dBD7W2PfvTQ42k+LmAdUFudTytO89oTRXcVEYMDL42OyRth12JRMUjYTEmFwo9a9Mb7cP8+bo7N2lV4iCB0CUybcZARF0MV6NeYhhWlC9DV40nkqs3Goe8X8tMPXn/HZn8Rz33703w8K/G6STnN0txhAT4tY7D3e0DA9UY87wNnpJ7dXoJqMXRv2dRgmUnGih/8cLHypyxBoLoL8qR9cWxAf/Cs+qQmsk15lzIGQUAJwwXBBjbnXKwykEeHjTHsvjd7zzC1cWtz5Zz/8aop7AsVwaBqb9u+5dVOMxdzLGD24NKTjhtG86ADU4Mpnlb5 > > > rob Updated patch attached. Honza -- Jan Cholasta -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-jcholast-83.1-openssh-style-public-keys.patch Type: text/x-patch Size: 34789 bytes Desc: not available URL: From rcritten at redhat.com Thu Sep 6 15:55:00 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 06 Sep 2012 11:55:00 -0400 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <5048C46A.8020207@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> <50479491.1060200@redhat.com> <50488DD0.1080507@redhat.com> <5048BEF8.5000401@redhat.com> <5048C46A.8020207@redhat.com> Message-ID: <5048C754.2030708@redhat.com> Rob Crittenden wrote: > Rob Crittenden wrote: >> Martin Kosek wrote: >>> On 09/05/2012 08:06 PM, Rob Crittenden wrote: >>>> Rob Crittenden wrote: >>>>> Martin Kosek wrote: >>>>>> On 07/05/2012 08:39 PM, Rob Crittenden wrote: >>>>>>> Martin Kosek wrote: >>>>>>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: >>>>>>>>> Deleting a replica can leave a replication vector (RUV) on the >>>>>>>>> other servers. >>>>>>>>> This can confuse things if the replica is re-added, and it also >>>>>>>>> causes the >>>>>>>>> server to calculate changes against a server that may no longer >>>>>>>>> exist. >>>>>>>>> >>>>>>>>> 389-ds-base provides a new task that self-propogates itself to all >>>>>>>>> available >>>>>>>>> replicas to clean this RUV data. >>>>>>>>> >>>>>>>>> This patch will create this task at deletion time to hopefully >>>>>>>>> clean things up. >>>>>>>>> >>>>>>>>> It isn't perfect. If any replica is down or unavailable at the >>>>>>>>> time >>>>>>>>> the >>>>>>>>> cleanruv task fires, and then comes back up, the old RUV data >>>>>>>>> may be >>>>>>>>> re-propogated around. >>>>>>>>> >>>>>>>>> To make things easier in this case I've added two new commands to >>>>>>>>> ipa-replica-manage. The first lists the replication ids of all the >>>>>>>>> servers we >>>>>>>>> have a RUV for. Using this you can call clean_ruv with the >>>>>>>>> replication id of a >>>>>>>>> server that no longer exists to try the cleanallruv step again. >>>>>>>>> >>>>>>>>> This is quite dangerous though. If you run cleanruv against a >>>>>>>>> replica id that >>>>>>>>> does exist it can cause a loss of data. I believe I've put in >>>>>>>>> enough scary >>>>>>>>> warnings about this. >>>>>>>>> >>>>>>>>> rob >>>>>>>>> >>>>>>>> >>>>>>>> Good work there, this should make cleaning RUVs much easier than >>>>>>>> with the >>>>>>>> previous version. >>>>>>>> >>>>>>>> This is what I found during review: >>>>>>>> >>>>>>>> 1) list_ruv and clean_ruv command help in man is quite lost. I >>>>>>>> think >>>>>>>> it would >>>>>>>> help if we for example have all info for commands indented. This >>>>>>>> way >>>>>>>> user could >>>>>>>> simply over-look the new commands in the man page. >>>>>>>> >>>>>>>> >>>>>>>> 2) I would rename new commands to clean-ruv and list-ruv to make >>>>>>>> them >>>>>>>> consistent with the rest of the commands (re-initialize, >>>>>>>> force-sync). >>>>>>>> >>>>>>>> >>>>>>>> 3) It would be nice to be able to run clean_ruv command in an >>>>>>>> unattended way >>>>>>>> (for better testing), i.e. respect --force option as we already >>>>>>>> do for >>>>>>>> ipa-replica-manage del. This fix would aid test automation in the >>>>>>>> future. >>>>>>>> >>>>>>>> >>>>>>>> 4) (minor) The new question (and the del too) does not react too >>>>>>>> well for >>>>>>>> CTRL+D: >>>>>>>> >>>>>>>> # ipa-replica-manage clean_ruv 3 --force >>>>>>>> Clean the Replication Update Vector for >>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>> >>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>> longer replicate so it may miss updates while the process >>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>> consistency. Be very careful. >>>>>>>> Continue to clean? [no]: unexpected error: >>>>>>>> >>>>>>>> >>>>>>>> 5) Help for clean_ruv command without a required parameter is quite >>>>>>>> confusing >>>>>>>> as it reports that command is wrong and not the parameter: >>>>>>>> >>>>>>>> # ipa-replica-manage clean_ruv >>>>>>>> Usage: ipa-replica-manage [options] >>>>>>>> >>>>>>>> ipa-replica-manage: error: must provide a command [clean_ruv | >>>>>>>> force-sync | >>>>>>>> disconnect | connect | del | re-initialize | list | list_ruv] >>>>>>>> >>>>>>>> It seems you just forgot to specify the error message in the >>>>>>>> command >>>>>>>> definition >>>>>>>> >>>>>>>> >>>>>>>> 6) When the remote replica is down, the clean_ruv command fails >>>>>>>> with an >>>>>>>> unexpected error: >>>>>>>> >>>>>>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 >>>>>>>> Clean the Replication Update Vector for >>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>> >>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>> longer replicate so it may miss updates while the process >>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>> consistency. Be very careful. >>>>>>>> Continue to clean? [no]: y >>>>>>>> unexpected error: {'desc': 'Operations error'} >>>>>>>> >>>>>>>> >>>>>>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: >>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>> cleanAllRUV_task: failed >>>>>>>> to connect to repl agreement connection >>>>>>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, >>>>>>>> >>>>>>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>> >>>>>>>> >>>>>>>> tree,cn=config), error 105 >>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>> cleanAllRUV_task: replica >>>>>>>> (cn=meTovm-055.idm.lab. >>>>>>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> tree, cn=config) has not been cleaned. You will need to rerun >>>>>>>> the >>>>>>>> CLEANALLRUV task on this replica. >>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>> cleanAllRUV_task: Task >>>>>>>> failed (1) >>>>>>>> >>>>>>>> In this case I think we should inform user that the command failed, >>>>>>>> possibly >>>>>>>> because of disconnected replicas and that they could enable the >>>>>>>> replicas and >>>>>>>> try again. >>>>>>>> >>>>>>>> >>>>>>>> 7) (minor) "pass" is now redundant in replication.py: >>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>> + # We can't make the server we're removing read-only >>>>>>>> but >>>>>>>> + # this isn't a show-stopper >>>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>>> read-only, >>>>>>>> continuing anyway") >>>>>>>> + pass >>>>>>>> >>>>>>> >>>>>>> I think this addresses everything. >>>>>>> >>>>>>> rob >>>>>> >>>>>> Thanks, almost there! I just found one more issue which needs to be >>>>>> fixed >>>>>> before we push: >>>>>> >>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>>>> Directory Manager password: >>>>>> >>>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing >>>>>> removal >>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>> "Can't >>>>>> contact LDAP server"} >>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>> >>>>>> There were issues removing a connection: %d format: a number is >>>>>> required, not str >>>>>> >>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>> "Can't >>>>>> contact LDAP server"} >>>>>> >>>>>> This is a traceback I retrieved: >>>>>> Traceback (most recent call last): >>>>>> File "/sbin/ipa-replica-manage", line 425, in del_master >>>>>> del_link(realm, r, hostname, options.dirman_passwd, force=True) >>>>>> File "/sbin/ipa-replica-manage", line 271, in del_link >>>>>> repl1.cleanallruv(replica_id) >>>>>> File >>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>>> line 1094, in cleanallruv >>>>>> root_logger.debug("Creating CLEANALLRUV task for replica id >>>>>> %d" % >>>>>> replicaId) >>>>>> >>>>>> >>>>>> The problem here is that you don't convert replica_id to int in this >>>>>> part: >>>>>> + replica_id = None >>>>>> + if repl2: >>>>>> + replica_id = repl2._get_replica_id(repl2.conn, None) >>>>>> + else: >>>>>> + servers = get_ruv(realm, replica1, dirman_passwd) >>>>>> + for (netloc, rid) in servers: >>>>>> + if netloc.startswith(replica2): >>>>>> + replica_id = rid >>>>>> + break >>>>>> >>>>>> Martin >>>>>> >>>>> >>>>> Updated patch using new mechanism in 389-ds-base. This should more >>>>> thoroughly clean out RUV data when a replica is being deleted, and >>>>> provide for a way to delete RUV data afterwards too if necessary. >>>>> >>>>> rob >>>> >>>> Rebased patch >>>> >>>> rob >>>> >>> >>> 0) As I wrote in a review for your patch 1041, changelog entry slipped >>> elsewhere. >>> >>> 1) The following KeyboardInterrupt except class looks suspicious. I >>> know why >>> you have it there, but since it is generally a bad thing to do, some >>> comment >>> why it is needed would be useful. >>> >>> @@ -256,6 +263,17 @@ def del_link(realm, replica1, replica2, >>> dirman_passwd, >>> force=False): >>> repl1.delete_agreement(replica2) >>> repl1.delete_referral(replica2) >>> >>> + if type1 == replication.IPA_REPLICA: >>> + if repl2: >>> + ruv = repl2._get_replica_id(repl2.conn, None) >>> + else: >>> + ruv = get_ruv_by_host(realm, replica1, replica2, >>> dirman_passwd) >>> + >>> + try: >>> + repl1.cleanallruv(ruv) >>> + except KeyboardInterrupt: >>> + pass >>> + >>> >>> Maybe you just wanted to do some cleanup and then "raise" again? >> >> No, it is there because it is safe to break out of it. The task will >> continue to run. I added some verbiage. >> >>> >>> 2) This is related to 1), but when some replica is down, >>> "ipa-replica-manage >>> del" may wait indefinitely when some remote replica is down, right? >>> >>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>> Deleting a master is irreversible. >>> To reconnect to the remote master you will need to prepare a new >>> replica file >>> and re-install. >>> Continue to delete? [no]: y >>> ipa: INFO: Setting agreement >>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>> >>> >>> tree,cn=config schedule to 2358-2359 0 to force synch >>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>> >>> >>> tree,cn=config >>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica >>> acquired >>> successfully: Incremental update succeeded: start: 0: end: 0 >>> Background task created to clean replication data >>> >>> ... after about a minute I hit CTRL+C >>> >>> ^CDeleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to >>> 'vm-055.idm.lab.bos.redhat.com' >>> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: NS record >>> does not >>> contain 'vm-055.idm.lab.bos.redhat.com.' >>> You may need to manually remove them from the tree >>> >>> I think it would be better to inform user that some remote replica is >>> down or >>> at least that we are waiting for the task to complete. Something like >>> that: >>> >>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>> ... >>> Background task created to clean replication data >>> Replication data clean up may take very long time if some replica is >>> unreachable >>> Hit CTRL+C to interrupt the wait >>> ^C Clean up wait interrupted >>> .... >>> [continue with del] >> >> Yup, did this in #1. >> >>> >>> 3) (minor) When there is a cleanruv task running and you run >>> "ipa-replica-manage del", there is a unexpected error message with >>> duplicate >>> task object in LDAP: >>> >>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com --force >>> Unable to connect to replica vm-072.idm.lab.bos.redhat.com, forcing >>> removal >>> FAIL >>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>> contact LDAP server"} >>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>> >>> There were issues removing a connection: This entry already exists >>> <<<<<<<<< >>> >>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>> contact LDAP server"} >>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record >>> does not >>> contain 'vm-072.idm.lab.bos.redhat.com.' >>> You may need to manually remove them from the tree >>> >>> >>> I think it should be enough to just catch for "entry already exists" in >>> cleanallruv function, and in such case print a relevant error message >>> bail out. >>> Thus, self.conn.checkTask(dn, dowait=True) would not be called too. >> >> Good catch, fixed. >> >>> >>> >>> 4) (minor): In make_readonly function, there is a redundant "pass" >>> statement: >>> >>> + def make_readonly(self): >>> + """ >>> + Make the current replication agreement read-only. >>> + """ >>> + dn = DN(('cn', 'userRoot'), ('cn', 'ldbm database'), >>> + ('cn', 'plugins'), ('cn', 'config')) >>> + >>> + mod = [(ldap.MOD_REPLACE, 'nsslapd-readonly', 'on')] >>> + try: >>> + self.conn.modify_s(dn, mod) >>> + except ldap.INSUFFICIENT_ACCESS: >>> + # We can't make the server we're removing read-only but >>> + # this isn't a show-stopper >>> + root_logger.debug("No permission to switch replica to >>> read-only, >>> continuing anyway") >>> + pass <<<<<<<<<<<<<<< >> >> Yeah, this is one of my common mistakes. I put in a pass initially, then >> add logging in front of it and forget to delete the pass. Its gone now. >> >>> >>> >>> 5) In clean_ruv, I think allowing a --force option to bypass the >>> user_input >>> would be helpful (at least for test automation): >>> >>> + if not ipautil.user_input("Continue to clean?", False): >>> + sys.exit("Aborted") >> >> Yup, added. >> >> rob > > Slightly revised patch. I still had a window open with one unsaved change. > > rob > Apparently there were two unsaved changes, one of which was lost. This adds in the 'entry already exists' fix. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-rcrit-1031-7-cleanruv.patch Type: text/x-diff Size: 14819 bytes Desc: not available URL: From sbose at redhat.com Thu Sep 6 15:55:19 2012 From: sbose at redhat.com (Sumit Bose) Date: Thu, 6 Sep 2012 17:55:19 +0200 Subject: [Freeipa-devel] [PATCH] 303 Add range safety check for range_mod and range_del In-Reply-To: <50476C25.9000009@redhat.com> References: <50476C25.9000009@redhat.com> Message-ID: <20120906155519.GA30746@localhost.localdomain> On Wed, Sep 05, 2012 at 05:13:41PM +0200, Martin Kosek wrote: > range_mod and range_del command could easily create objects with > ID which is suddenly out of specified range. This could cause issues > in trust scenarios where range objects are used for computation of > remote IDs. > > Add validator for both commands to check if there is any object with > ID in the range which would become out-of-range as a pre_callback. > Also add unit tests testing this new validator. > > https://fedorahosted.org/freeipa/ticket/2919 ACK bye, Sumit From mkosek at redhat.com Thu Sep 6 16:05:18 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 06 Sep 2012 18:05:18 +0200 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <5048C754.2030708@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> <50479491.1060200@redhat.com> <50488DD0.1080507@redhat.com> <5048BEF8.5000401@redhat.com> <5048C46A.8020207@redhat.com> <5048C754.2030708@redhat.com> Message-ID: <5048C9BE.1050005@redhat.com> On 09/06/2012 05:55 PM, Rob Crittenden wrote: > Rob Crittenden wrote: >> Rob Crittenden wrote: >>> Martin Kosek wrote: >>>> On 09/05/2012 08:06 PM, Rob Crittenden wrote: >>>>> Rob Crittenden wrote: >>>>>> Martin Kosek wrote: >>>>>>> On 07/05/2012 08:39 PM, Rob Crittenden wrote: >>>>>>>> Martin Kosek wrote: >>>>>>>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: >>>>>>>>>> Deleting a replica can leave a replication vector (RUV) on the >>>>>>>>>> other servers. >>>>>>>>>> This can confuse things if the replica is re-added, and it also >>>>>>>>>> causes the >>>>>>>>>> server to calculate changes against a server that may no longer >>>>>>>>>> exist. >>>>>>>>>> >>>>>>>>>> 389-ds-base provides a new task that self-propogates itself to all >>>>>>>>>> available >>>>>>>>>> replicas to clean this RUV data. >>>>>>>>>> >>>>>>>>>> This patch will create this task at deletion time to hopefully >>>>>>>>>> clean things up. >>>>>>>>>> >>>>>>>>>> It isn't perfect. If any replica is down or unavailable at the >>>>>>>>>> time >>>>>>>>>> the >>>>>>>>>> cleanruv task fires, and then comes back up, the old RUV data >>>>>>>>>> may be >>>>>>>>>> re-propogated around. >>>>>>>>>> >>>>>>>>>> To make things easier in this case I've added two new commands to >>>>>>>>>> ipa-replica-manage. The first lists the replication ids of all the >>>>>>>>>> servers we >>>>>>>>>> have a RUV for. Using this you can call clean_ruv with the >>>>>>>>>> replication id of a >>>>>>>>>> server that no longer exists to try the cleanallruv step again. >>>>>>>>>> >>>>>>>>>> This is quite dangerous though. If you run cleanruv against a >>>>>>>>>> replica id that >>>>>>>>>> does exist it can cause a loss of data. I believe I've put in >>>>>>>>>> enough scary >>>>>>>>>> warnings about this. >>>>>>>>>> >>>>>>>>>> rob >>>>>>>>>> >>>>>>>>> >>>>>>>>> Good work there, this should make cleaning RUVs much easier than >>>>>>>>> with the >>>>>>>>> previous version. >>>>>>>>> >>>>>>>>> This is what I found during review: >>>>>>>>> >>>>>>>>> 1) list_ruv and clean_ruv command help in man is quite lost. I >>>>>>>>> think >>>>>>>>> it would >>>>>>>>> help if we for example have all info for commands indented. This >>>>>>>>> way >>>>>>>>> user could >>>>>>>>> simply over-look the new commands in the man page. >>>>>>>>> >>>>>>>>> >>>>>>>>> 2) I would rename new commands to clean-ruv and list-ruv to make >>>>>>>>> them >>>>>>>>> consistent with the rest of the commands (re-initialize, >>>>>>>>> force-sync). >>>>>>>>> >>>>>>>>> >>>>>>>>> 3) It would be nice to be able to run clean_ruv command in an >>>>>>>>> unattended way >>>>>>>>> (for better testing), i.e. respect --force option as we already >>>>>>>>> do for >>>>>>>>> ipa-replica-manage del. This fix would aid test automation in the >>>>>>>>> future. >>>>>>>>> >>>>>>>>> >>>>>>>>> 4) (minor) The new question (and the del too) does not react too >>>>>>>>> well for >>>>>>>>> CTRL+D: >>>>>>>>> >>>>>>>>> # ipa-replica-manage clean_ruv 3 --force >>>>>>>>> Clean the Replication Update Vector for >>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>> >>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>> consistency. Be very careful. >>>>>>>>> Continue to clean? [no]: unexpected error: >>>>>>>>> >>>>>>>>> >>>>>>>>> 5) Help for clean_ruv command without a required parameter is quite >>>>>>>>> confusing >>>>>>>>> as it reports that command is wrong and not the parameter: >>>>>>>>> >>>>>>>>> # ipa-replica-manage clean_ruv >>>>>>>>> Usage: ipa-replica-manage [options] >>>>>>>>> >>>>>>>>> ipa-replica-manage: error: must provide a command [clean_ruv | >>>>>>>>> force-sync | >>>>>>>>> disconnect | connect | del | re-initialize | list | list_ruv] >>>>>>>>> >>>>>>>>> It seems you just forgot to specify the error message in the >>>>>>>>> command >>>>>>>>> definition >>>>>>>>> >>>>>>>>> >>>>>>>>> 6) When the remote replica is down, the clean_ruv command fails >>>>>>>>> with an >>>>>>>>> unexpected error: >>>>>>>>> >>>>>>>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 >>>>>>>>> Clean the Replication Update Vector for >>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>> >>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>> consistency. Be very careful. >>>>>>>>> Continue to clean? [no]: y >>>>>>>>> unexpected error: {'desc': 'Operations error'} >>>>>>>>> >>>>>>>>> >>>>>>>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: >>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>> cleanAllRUV_task: failed >>>>>>>>> to connect to repl agreement connection >>>>>>>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, >>>>>>>>> >>>>>>>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>> >>>>>>>>> >>>>>>>>> tree,cn=config), error 105 >>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>> cleanAllRUV_task: replica >>>>>>>>> (cn=meTovm-055.idm.lab. >>>>>>>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> tree, cn=config) has not been cleaned. You will need to rerun >>>>>>>>> the >>>>>>>>> CLEANALLRUV task on this replica. >>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>> cleanAllRUV_task: Task >>>>>>>>> failed (1) >>>>>>>>> >>>>>>>>> In this case I think we should inform user that the command failed, >>>>>>>>> possibly >>>>>>>>> because of disconnected replicas and that they could enable the >>>>>>>>> replicas and >>>>>>>>> try again. >>>>>>>>> >>>>>>>>> >>>>>>>>> 7) (minor) "pass" is now redundant in replication.py: >>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>>> + # We can't make the server we're removing read-only >>>>>>>>> but >>>>>>>>> + # this isn't a show-stopper >>>>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>>>> read-only, >>>>>>>>> continuing anyway") >>>>>>>>> + pass >>>>>>>>> >>>>>>>> >>>>>>>> I think this addresses everything. >>>>>>>> >>>>>>>> rob >>>>>>> >>>>>>> Thanks, almost there! I just found one more issue which needs to be >>>>>>> fixed >>>>>>> before we push: >>>>>>> >>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>>>>> Directory Manager password: >>>>>>> >>>>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing >>>>>>> removal >>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>> "Can't >>>>>>> contact LDAP server"} >>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>> >>>>>>> There were issues removing a connection: %d format: a number is >>>>>>> required, not str >>>>>>> >>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>> "Can't >>>>>>> contact LDAP server"} >>>>>>> >>>>>>> This is a traceback I retrieved: >>>>>>> Traceback (most recent call last): >>>>>>> File "/sbin/ipa-replica-manage", line 425, in del_master >>>>>>> del_link(realm, r, hostname, options.dirman_passwd, force=True) >>>>>>> File "/sbin/ipa-replica-manage", line 271, in del_link >>>>>>> repl1.cleanallruv(replica_id) >>>>>>> File >>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>>>> line 1094, in cleanallruv >>>>>>> root_logger.debug("Creating CLEANALLRUV task for replica id >>>>>>> %d" % >>>>>>> replicaId) >>>>>>> >>>>>>> >>>>>>> The problem here is that you don't convert replica_id to int in this >>>>>>> part: >>>>>>> + replica_id = None >>>>>>> + if repl2: >>>>>>> + replica_id = repl2._get_replica_id(repl2.conn, None) >>>>>>> + else: >>>>>>> + servers = get_ruv(realm, replica1, dirman_passwd) >>>>>>> + for (netloc, rid) in servers: >>>>>>> + if netloc.startswith(replica2): >>>>>>> + replica_id = rid >>>>>>> + break >>>>>>> >>>>>>> Martin >>>>>>> >>>>>> >>>>>> Updated patch using new mechanism in 389-ds-base. This should more >>>>>> thoroughly clean out RUV data when a replica is being deleted, and >>>>>> provide for a way to delete RUV data afterwards too if necessary. >>>>>> >>>>>> rob >>>>> >>>>> Rebased patch >>>>> >>>>> rob >>>>> >>>> >>>> 0) As I wrote in a review for your patch 1041, changelog entry slipped >>>> elsewhere. >>>> >>>> 1) The following KeyboardInterrupt except class looks suspicious. I >>>> know why >>>> you have it there, but since it is generally a bad thing to do, some >>>> comment >>>> why it is needed would be useful. >>>> >>>> @@ -256,6 +263,17 @@ def del_link(realm, replica1, replica2, >>>> dirman_passwd, >>>> force=False): >>>> repl1.delete_agreement(replica2) >>>> repl1.delete_referral(replica2) >>>> >>>> + if type1 == replication.IPA_REPLICA: >>>> + if repl2: >>>> + ruv = repl2._get_replica_id(repl2.conn, None) >>>> + else: >>>> + ruv = get_ruv_by_host(realm, replica1, replica2, >>>> dirman_passwd) >>>> + >>>> + try: >>>> + repl1.cleanallruv(ruv) >>>> + except KeyboardInterrupt: >>>> + pass >>>> + >>>> >>>> Maybe you just wanted to do some cleanup and then "raise" again? >>> >>> No, it is there because it is safe to break out of it. The task will >>> continue to run. I added some verbiage. >>> >>>> >>>> 2) This is related to 1), but when some replica is down, >>>> "ipa-replica-manage >>>> del" may wait indefinitely when some remote replica is down, right? >>>> >>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>> Deleting a master is irreversible. >>>> To reconnect to the remote master you will need to prepare a new >>>> replica file >>>> and re-install. >>>> Continue to delete? [no]: y >>>> ipa: INFO: Setting agreement >>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>> >>>> >>>> >>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>> >>>> >>>> >>>> tree,cn=config >>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica >>>> acquired >>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>> Background task created to clean replication data >>>> >>>> ... after about a minute I hit CTRL+C >>>> >>>> ^CDeleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to >>>> 'vm-055.idm.lab.bos.redhat.com' >>>> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: NS record >>>> does not >>>> contain 'vm-055.idm.lab.bos.redhat.com.' >>>> You may need to manually remove them from the tree >>>> >>>> I think it would be better to inform user that some remote replica is >>>> down or >>>> at least that we are waiting for the task to complete. Something like >>>> that: >>>> >>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>> ... >>>> Background task created to clean replication data >>>> Replication data clean up may take very long time if some replica is >>>> unreachable >>>> Hit CTRL+C to interrupt the wait >>>> ^C Clean up wait interrupted >>>> .... >>>> [continue with del] >>> >>> Yup, did this in #1. >>> >>>> >>>> 3) (minor) When there is a cleanruv task running and you run >>>> "ipa-replica-manage del", there is a unexpected error message with >>>> duplicate >>>> task object in LDAP: >>>> >>>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com --force >>>> Unable to connect to replica vm-072.idm.lab.bos.redhat.com, forcing >>>> removal >>>> FAIL >>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>> contact LDAP server"} >>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>> >>>> There were issues removing a connection: This entry already exists >>>> <<<<<<<<< >>>> >>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>> contact LDAP server"} >>>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record >>>> does not >>>> contain 'vm-072.idm.lab.bos.redhat.com.' >>>> You may need to manually remove them from the tree >>>> >>>> >>>> I think it should be enough to just catch for "entry already exists" in >>>> cleanallruv function, and in such case print a relevant error message >>>> bail out. >>>> Thus, self.conn.checkTask(dn, dowait=True) would not be called too. >>> >>> Good catch, fixed. >>> >>>> >>>> >>>> 4) (minor): In make_readonly function, there is a redundant "pass" >>>> statement: >>>> >>>> + def make_readonly(self): >>>> + """ >>>> + Make the current replication agreement read-only. >>>> + """ >>>> + dn = DN(('cn', 'userRoot'), ('cn', 'ldbm database'), >>>> + ('cn', 'plugins'), ('cn', 'config')) >>>> + >>>> + mod = [(ldap.MOD_REPLACE, 'nsslapd-readonly', 'on')] >>>> + try: >>>> + self.conn.modify_s(dn, mod) >>>> + except ldap.INSUFFICIENT_ACCESS: >>>> + # We can't make the server we're removing read-only but >>>> + # this isn't a show-stopper >>>> + root_logger.debug("No permission to switch replica to >>>> read-only, >>>> continuing anyway") >>>> + pass <<<<<<<<<<<<<<< >>> >>> Yeah, this is one of my common mistakes. I put in a pass initially, then >>> add logging in front of it and forget to delete the pass. Its gone now. >>> >>>> >>>> >>>> 5) In clean_ruv, I think allowing a --force option to bypass the >>>> user_input >>>> would be helpful (at least for test automation): >>>> >>>> + if not ipautil.user_input("Continue to clean?", False): >>>> + sys.exit("Aborted") >>> >>> Yup, added. >>> >>> rob >> >> Slightly revised patch. I still had a window open with one unsaved change. >> >> rob >> > > Apparently there were two unsaved changes, one of which was lost. This adds in > the 'entry already exists' fix. > > rob > Just one last thing (otherwise the patch is OK) - I don't think this is what we want :-) # ipa-replica-manage clean-ruv 8 Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 Cleaning the wrong replica ID will cause that server to no longer replicate so it may miss updates while the process is running. It would need to be re-initialized to maintain consistency. Be very careful. Continue to clean? [no]: y <<<<<< Aborted Nor this exception, (your are checking for wrong exception): # ipa-replica-manage clean-ruv 8 Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 Cleaning the wrong replica ID will cause that server to no longer replicate so it may miss updates while the process is running. It would need to be re-initialized to maintain consistency. Be very careful. Continue to clean? [no]: unexpected error: This entry already exists This is the exception: Traceback (most recent call last): File "/sbin/ipa-replica-manage", line 651, in main() File "/sbin/ipa-replica-manage", line 648, in main clean_ruv(realm, args[1], options) File "/sbin/ipa-replica-manage", line 373, in clean_ruv thisrepl.cleanallruv(ruv) File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", line 1136, in cleanallruv self.conn.addEntry(e) File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 503, in addEntry self.__handle_errors(e, arg_desc=arg_desc) File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 321, in __handle_errors raise errors.DuplicateEntry() ipalib.errors.DuplicateEntry: This entry already exists Martin From mkosek at redhat.com Thu Sep 6 16:09:03 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 06 Sep 2012 18:09:03 +0200 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <5048C9BE.1050005@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> <50479491.1060200@redhat.com> <50488DD0.1080507@redhat.com> <5048BEF8.5000401@redhat.com> <5048C46A.8020207@redhat.com> <5048C754.2030708@redhat.com> <5048C9BE.1050005@redhat.com> Message-ID: <5048CA9F.7060607@redhat.com> On 09/06/2012 06:05 PM, Martin Kosek wrote: > On 09/06/2012 05:55 PM, Rob Crittenden wrote: >> Rob Crittenden wrote: >>> Rob Crittenden wrote: >>>> Martin Kosek wrote: >>>>> On 09/05/2012 08:06 PM, Rob Crittenden wrote: >>>>>> Rob Crittenden wrote: >>>>>>> Martin Kosek wrote: >>>>>>>> On 07/05/2012 08:39 PM, Rob Crittenden wrote: >>>>>>>>> Martin Kosek wrote: >>>>>>>>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: >>>>>>>>>>> Deleting a replica can leave a replication vector (RUV) on the >>>>>>>>>>> other servers. >>>>>>>>>>> This can confuse things if the replica is re-added, and it also >>>>>>>>>>> causes the >>>>>>>>>>> server to calculate changes against a server that may no longer >>>>>>>>>>> exist. >>>>>>>>>>> >>>>>>>>>>> 389-ds-base provides a new task that self-propogates itself to all >>>>>>>>>>> available >>>>>>>>>>> replicas to clean this RUV data. >>>>>>>>>>> >>>>>>>>>>> This patch will create this task at deletion time to hopefully >>>>>>>>>>> clean things up. >>>>>>>>>>> >>>>>>>>>>> It isn't perfect. If any replica is down or unavailable at the >>>>>>>>>>> time >>>>>>>>>>> the >>>>>>>>>>> cleanruv task fires, and then comes back up, the old RUV data >>>>>>>>>>> may be >>>>>>>>>>> re-propogated around. >>>>>>>>>>> >>>>>>>>>>> To make things easier in this case I've added two new commands to >>>>>>>>>>> ipa-replica-manage. The first lists the replication ids of all the >>>>>>>>>>> servers we >>>>>>>>>>> have a RUV for. Using this you can call clean_ruv with the >>>>>>>>>>> replication id of a >>>>>>>>>>> server that no longer exists to try the cleanallruv step again. >>>>>>>>>>> >>>>>>>>>>> This is quite dangerous though. If you run cleanruv against a >>>>>>>>>>> replica id that >>>>>>>>>>> does exist it can cause a loss of data. I believe I've put in >>>>>>>>>>> enough scary >>>>>>>>>>> warnings about this. >>>>>>>>>>> >>>>>>>>>>> rob >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Good work there, this should make cleaning RUVs much easier than >>>>>>>>>> with the >>>>>>>>>> previous version. >>>>>>>>>> >>>>>>>>>> This is what I found during review: >>>>>>>>>> >>>>>>>>>> 1) list_ruv and clean_ruv command help in man is quite lost. I >>>>>>>>>> think >>>>>>>>>> it would >>>>>>>>>> help if we for example have all info for commands indented. This >>>>>>>>>> way >>>>>>>>>> user could >>>>>>>>>> simply over-look the new commands in the man page. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2) I would rename new commands to clean-ruv and list-ruv to make >>>>>>>>>> them >>>>>>>>>> consistent with the rest of the commands (re-initialize, >>>>>>>>>> force-sync). >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 3) It would be nice to be able to run clean_ruv command in an >>>>>>>>>> unattended way >>>>>>>>>> (for better testing), i.e. respect --force option as we already >>>>>>>>>> do for >>>>>>>>>> ipa-replica-manage del. This fix would aid test automation in the >>>>>>>>>> future. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 4) (minor) The new question (and the del too) does not react too >>>>>>>>>> well for >>>>>>>>>> CTRL+D: >>>>>>>>>> >>>>>>>>>> # ipa-replica-manage clean_ruv 3 --force >>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>> >>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>> consistency. Be very careful. >>>>>>>>>> Continue to clean? [no]: unexpected error: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 5) Help for clean_ruv command without a required parameter is quite >>>>>>>>>> confusing >>>>>>>>>> as it reports that command is wrong and not the parameter: >>>>>>>>>> >>>>>>>>>> # ipa-replica-manage clean_ruv >>>>>>>>>> Usage: ipa-replica-manage [options] >>>>>>>>>> >>>>>>>>>> ipa-replica-manage: error: must provide a command [clean_ruv | >>>>>>>>>> force-sync | >>>>>>>>>> disconnect | connect | del | re-initialize | list | list_ruv] >>>>>>>>>> >>>>>>>>>> It seems you just forgot to specify the error message in the >>>>>>>>>> command >>>>>>>>>> definition >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 6) When the remote replica is down, the clean_ruv command fails >>>>>>>>>> with an >>>>>>>>>> unexpected error: >>>>>>>>>> >>>>>>>>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 >>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>> >>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>> consistency. Be very careful. >>>>>>>>>> Continue to clean? [no]: y >>>>>>>>>> unexpected error: {'desc': 'Operations error'} >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: >>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>> cleanAllRUV_task: failed >>>>>>>>>> to connect to repl agreement connection >>>>>>>>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, >>>>>>>>>> >>>>>>>>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> tree,cn=config), error 105 >>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>> cleanAllRUV_task: replica >>>>>>>>>> (cn=meTovm-055.idm.lab. >>>>>>>>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> tree, cn=config) has not been cleaned. You will need to rerun >>>>>>>>>> the >>>>>>>>>> CLEANALLRUV task on this replica. >>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>> cleanAllRUV_task: Task >>>>>>>>>> failed (1) >>>>>>>>>> >>>>>>>>>> In this case I think we should inform user that the command failed, >>>>>>>>>> possibly >>>>>>>>>> because of disconnected replicas and that they could enable the >>>>>>>>>> replicas and >>>>>>>>>> try again. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 7) (minor) "pass" is now redundant in replication.py: >>>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>>>> + # We can't make the server we're removing read-only >>>>>>>>>> but >>>>>>>>>> + # this isn't a show-stopper >>>>>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>>>>> read-only, >>>>>>>>>> continuing anyway") >>>>>>>>>> + pass >>>>>>>>>> >>>>>>>>> >>>>>>>>> I think this addresses everything. >>>>>>>>> >>>>>>>>> rob >>>>>>>> >>>>>>>> Thanks, almost there! I just found one more issue which needs to be >>>>>>>> fixed >>>>>>>> before we push: >>>>>>>> >>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>>>>>> Directory Manager password: >>>>>>>> >>>>>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing >>>>>>>> removal >>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>> "Can't >>>>>>>> contact LDAP server"} >>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>> >>>>>>>> There were issues removing a connection: %d format: a number is >>>>>>>> required, not str >>>>>>>> >>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>> "Can't >>>>>>>> contact LDAP server"} >>>>>>>> >>>>>>>> This is a traceback I retrieved: >>>>>>>> Traceback (most recent call last): >>>>>>>> File "/sbin/ipa-replica-manage", line 425, in del_master >>>>>>>> del_link(realm, r, hostname, options.dirman_passwd, force=True) >>>>>>>> File "/sbin/ipa-replica-manage", line 271, in del_link >>>>>>>> repl1.cleanallruv(replica_id) >>>>>>>> File >>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>>>>> line 1094, in cleanallruv >>>>>>>> root_logger.debug("Creating CLEANALLRUV task for replica id >>>>>>>> %d" % >>>>>>>> replicaId) >>>>>>>> >>>>>>>> >>>>>>>> The problem here is that you don't convert replica_id to int in this >>>>>>>> part: >>>>>>>> + replica_id = None >>>>>>>> + if repl2: >>>>>>>> + replica_id = repl2._get_replica_id(repl2.conn, None) >>>>>>>> + else: >>>>>>>> + servers = get_ruv(realm, replica1, dirman_passwd) >>>>>>>> + for (netloc, rid) in servers: >>>>>>>> + if netloc.startswith(replica2): >>>>>>>> + replica_id = rid >>>>>>>> + break >>>>>>>> >>>>>>>> Martin >>>>>>>> >>>>>>> >>>>>>> Updated patch using new mechanism in 389-ds-base. This should more >>>>>>> thoroughly clean out RUV data when a replica is being deleted, and >>>>>>> provide for a way to delete RUV data afterwards too if necessary. >>>>>>> >>>>>>> rob >>>>>> >>>>>> Rebased patch >>>>>> >>>>>> rob >>>>>> >>>>> >>>>> 0) As I wrote in a review for your patch 1041, changelog entry slipped >>>>> elsewhere. >>>>> >>>>> 1) The following KeyboardInterrupt except class looks suspicious. I >>>>> know why >>>>> you have it there, but since it is generally a bad thing to do, some >>>>> comment >>>>> why it is needed would be useful. >>>>> >>>>> @@ -256,6 +263,17 @@ def del_link(realm, replica1, replica2, >>>>> dirman_passwd, >>>>> force=False): >>>>> repl1.delete_agreement(replica2) >>>>> repl1.delete_referral(replica2) >>>>> >>>>> + if type1 == replication.IPA_REPLICA: >>>>> + if repl2: >>>>> + ruv = repl2._get_replica_id(repl2.conn, None) >>>>> + else: >>>>> + ruv = get_ruv_by_host(realm, replica1, replica2, >>>>> dirman_passwd) >>>>> + >>>>> + try: >>>>> + repl1.cleanallruv(ruv) >>>>> + except KeyboardInterrupt: >>>>> + pass >>>>> + >>>>> >>>>> Maybe you just wanted to do some cleanup and then "raise" again? >>>> >>>> No, it is there because it is safe to break out of it. The task will >>>> continue to run. I added some verbiage. >>>> >>>>> >>>>> 2) This is related to 1), but when some replica is down, >>>>> "ipa-replica-manage >>>>> del" may wait indefinitely when some remote replica is down, right? >>>>> >>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>> Deleting a master is irreversible. >>>>> To reconnect to the remote master you will need to prepare a new >>>>> replica file >>>>> and re-install. >>>>> Continue to delete? [no]: y >>>>> ipa: INFO: Setting agreement >>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>> >>>>> >>>>> >>>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>> >>>>> >>>>> >>>>> tree,cn=config >>>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica >>>>> acquired >>>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>>> Background task created to clean replication data >>>>> >>>>> ... after about a minute I hit CTRL+C >>>>> >>>>> ^CDeleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to >>>>> 'vm-055.idm.lab.bos.redhat.com' >>>>> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: NS record >>>>> does not >>>>> contain 'vm-055.idm.lab.bos.redhat.com.' >>>>> You may need to manually remove them from the tree >>>>> >>>>> I think it would be better to inform user that some remote replica is >>>>> down or >>>>> at least that we are waiting for the task to complete. Something like >>>>> that: >>>>> >>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>> ... >>>>> Background task created to clean replication data >>>>> Replication data clean up may take very long time if some replica is >>>>> unreachable >>>>> Hit CTRL+C to interrupt the wait >>>>> ^C Clean up wait interrupted >>>>> .... >>>>> [continue with del] >>>> >>>> Yup, did this in #1. >>>> >>>>> >>>>> 3) (minor) When there is a cleanruv task running and you run >>>>> "ipa-replica-manage del", there is a unexpected error message with >>>>> duplicate >>>>> task object in LDAP: >>>>> >>>>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com --force >>>>> Unable to connect to replica vm-072.idm.lab.bos.redhat.com, forcing >>>>> removal >>>>> FAIL >>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>> contact LDAP server"} >>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>> >>>>> There were issues removing a connection: This entry already exists >>>>> <<<<<<<<< >>>>> >>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>> contact LDAP server"} >>>>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record >>>>> does not >>>>> contain 'vm-072.idm.lab.bos.redhat.com.' >>>>> You may need to manually remove them from the tree >>>>> >>>>> >>>>> I think it should be enough to just catch for "entry already exists" in >>>>> cleanallruv function, and in such case print a relevant error message >>>>> bail out. >>>>> Thus, self.conn.checkTask(dn, dowait=True) would not be called too. >>>> >>>> Good catch, fixed. >>>> >>>>> >>>>> >>>>> 4) (minor): In make_readonly function, there is a redundant "pass" >>>>> statement: >>>>> >>>>> + def make_readonly(self): >>>>> + """ >>>>> + Make the current replication agreement read-only. >>>>> + """ >>>>> + dn = DN(('cn', 'userRoot'), ('cn', 'ldbm database'), >>>>> + ('cn', 'plugins'), ('cn', 'config')) >>>>> + >>>>> + mod = [(ldap.MOD_REPLACE, 'nsslapd-readonly', 'on')] >>>>> + try: >>>>> + self.conn.modify_s(dn, mod) >>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>> + # We can't make the server we're removing read-only but >>>>> + # this isn't a show-stopper >>>>> + root_logger.debug("No permission to switch replica to >>>>> read-only, >>>>> continuing anyway") >>>>> + pass <<<<<<<<<<<<<<< >>>> >>>> Yeah, this is one of my common mistakes. I put in a pass initially, then >>>> add logging in front of it and forget to delete the pass. Its gone now. >>>> >>>>> >>>>> >>>>> 5) In clean_ruv, I think allowing a --force option to bypass the >>>>> user_input >>>>> would be helpful (at least for test automation): >>>>> >>>>> + if not ipautil.user_input("Continue to clean?", False): >>>>> + sys.exit("Aborted") >>>> >>>> Yup, added. >>>> >>>> rob >>> >>> Slightly revised patch. I still had a window open with one unsaved change. >>> >>> rob >>> >> >> Apparently there were two unsaved changes, one of which was lost. This adds in >> the 'entry already exists' fix. >> >> rob >> > > Just one last thing (otherwise the patch is OK) - I don't think this is what we > want :-) > > # ipa-replica-manage clean-ruv 8 > Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 > > Cleaning the wrong replica ID will cause that server to no > longer replicate so it may miss updates while the process > is running. It would need to be re-initialized to maintain > consistency. Be very careful. > Continue to clean? [no]: y <<<<<< > Aborted > > > Nor this exception, (your are checking for wrong exception): > > # ipa-replica-manage clean-ruv 8 > Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 > > Cleaning the wrong replica ID will cause that server to no > longer replicate so it may miss updates while the process > is running. It would need to be re-initialized to maintain > consistency. Be very careful. > Continue to clean? [no]: > unexpected error: This entry already exists > > This is the exception: > > Traceback (most recent call last): > File "/sbin/ipa-replica-manage", line 651, in > main() > File "/sbin/ipa-replica-manage", line 648, in main > clean_ruv(realm, args[1], options) > File "/sbin/ipa-replica-manage", line 373, in clean_ruv > thisrepl.cleanallruv(ruv) > File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", > line 1136, in cleanallruv > self.conn.addEntry(e) > File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 503, in > addEntry > self.__handle_errors(e, arg_desc=arg_desc) > File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 321, in > __handle_errors > raise errors.DuplicateEntry() > ipalib.errors.DuplicateEntry: This entry already exists > > Martin > On another matter, I just noticed that CLEANRUV is not proceeding if I have a winsync replica defined (and it is even up): # ipa-replica-manage list dc.ad.test: winsync <<<<<<< vm-072.idm.lab.bos.redhat.com: master vm-086.idm.lab.bos.redhat.com: master [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Waiting for all the replicas to receive all the deleted replica updates... [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Failed to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), will retry later. [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas caught up, retrying in 10 seconds [06/Sep/2012:11:59:20 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Failed to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), will retry later. [06/Sep/2012:11:59:20 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas caught up, retrying in 20 seconds [06/Sep/2012:11:59:40 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Failed to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), will retry later. [06/Sep/2012:11:59:40 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas caught up, retrying in 40 seconds I don't think this is OK. Adding Rich to CC to follow on this one. Martin From mkosek at redhat.com Thu Sep 6 16:09:52 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 06 Sep 2012 18:09:52 +0200 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <5048CA9F.7060607@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> <50479491.1060200@redhat.com> <50488DD0.1080507@redhat.com> <5048BEF8.5000401@redhat.com> <5048C46A.8020207@redhat.com> <5048C754.2030708@redhat.com> <5048C9BE.1050005@redhat.com> <5048CA9F.7060607@redhat.com> Message-ID: <5048CAD0.2070508@redhat.com> On 09/06/2012 06:09 PM, Martin Kosek wrote: > On 09/06/2012 06:05 PM, Martin Kosek wrote: >> On 09/06/2012 05:55 PM, Rob Crittenden wrote: >>> Rob Crittenden wrote: >>>> Rob Crittenden wrote: >>>>> Martin Kosek wrote: >>>>>> On 09/05/2012 08:06 PM, Rob Crittenden wrote: >>>>>>> Rob Crittenden wrote: >>>>>>>> Martin Kosek wrote: >>>>>>>>> On 07/05/2012 08:39 PM, Rob Crittenden wrote: >>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: >>>>>>>>>>>> Deleting a replica can leave a replication vector (RUV) on the >>>>>>>>>>>> other servers. >>>>>>>>>>>> This can confuse things if the replica is re-added, and it also >>>>>>>>>>>> causes the >>>>>>>>>>>> server to calculate changes against a server that may no longer >>>>>>>>>>>> exist. >>>>>>>>>>>> >>>>>>>>>>>> 389-ds-base provides a new task that self-propogates itself to all >>>>>>>>>>>> available >>>>>>>>>>>> replicas to clean this RUV data. >>>>>>>>>>>> >>>>>>>>>>>> This patch will create this task at deletion time to hopefully >>>>>>>>>>>> clean things up. >>>>>>>>>>>> >>>>>>>>>>>> It isn't perfect. If any replica is down or unavailable at the >>>>>>>>>>>> time >>>>>>>>>>>> the >>>>>>>>>>>> cleanruv task fires, and then comes back up, the old RUV data >>>>>>>>>>>> may be >>>>>>>>>>>> re-propogated around. >>>>>>>>>>>> >>>>>>>>>>>> To make things easier in this case I've added two new commands to >>>>>>>>>>>> ipa-replica-manage. The first lists the replication ids of all the >>>>>>>>>>>> servers we >>>>>>>>>>>> have a RUV for. Using this you can call clean_ruv with the >>>>>>>>>>>> replication id of a >>>>>>>>>>>> server that no longer exists to try the cleanallruv step again. >>>>>>>>>>>> >>>>>>>>>>>> This is quite dangerous though. If you run cleanruv against a >>>>>>>>>>>> replica id that >>>>>>>>>>>> does exist it can cause a loss of data. I believe I've put in >>>>>>>>>>>> enough scary >>>>>>>>>>>> warnings about this. >>>>>>>>>>>> >>>>>>>>>>>> rob >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Good work there, this should make cleaning RUVs much easier than >>>>>>>>>>> with the >>>>>>>>>>> previous version. >>>>>>>>>>> >>>>>>>>>>> This is what I found during review: >>>>>>>>>>> >>>>>>>>>>> 1) list_ruv and clean_ruv command help in man is quite lost. I >>>>>>>>>>> think >>>>>>>>>>> it would >>>>>>>>>>> help if we for example have all info for commands indented. This >>>>>>>>>>> way >>>>>>>>>>> user could >>>>>>>>>>> simply over-look the new commands in the man page. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2) I would rename new commands to clean-ruv and list-ruv to make >>>>>>>>>>> them >>>>>>>>>>> consistent with the rest of the commands (re-initialize, >>>>>>>>>>> force-sync). >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 3) It would be nice to be able to run clean_ruv command in an >>>>>>>>>>> unattended way >>>>>>>>>>> (for better testing), i.e. respect --force option as we already >>>>>>>>>>> do for >>>>>>>>>>> ipa-replica-manage del. This fix would aid test automation in the >>>>>>>>>>> future. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 4) (minor) The new question (and the del too) does not react too >>>>>>>>>>> well for >>>>>>>>>>> CTRL+D: >>>>>>>>>>> >>>>>>>>>>> # ipa-replica-manage clean_ruv 3 --force >>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>> >>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>> Continue to clean? [no]: unexpected error: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 5) Help for clean_ruv command without a required parameter is quite >>>>>>>>>>> confusing >>>>>>>>>>> as it reports that command is wrong and not the parameter: >>>>>>>>>>> >>>>>>>>>>> # ipa-replica-manage clean_ruv >>>>>>>>>>> Usage: ipa-replica-manage [options] >>>>>>>>>>> >>>>>>>>>>> ipa-replica-manage: error: must provide a command [clean_ruv | >>>>>>>>>>> force-sync | >>>>>>>>>>> disconnect | connect | del | re-initialize | list | list_ruv] >>>>>>>>>>> >>>>>>>>>>> It seems you just forgot to specify the error message in the >>>>>>>>>>> command >>>>>>>>>>> definition >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 6) When the remote replica is down, the clean_ruv command fails >>>>>>>>>>> with an >>>>>>>>>>> unexpected error: >>>>>>>>>>> >>>>>>>>>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 >>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>> >>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>> Continue to clean? [no]: y >>>>>>>>>>> unexpected error: {'desc': 'Operations error'} >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: >>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>> cleanAllRUV_task: failed >>>>>>>>>>> to connect to repl agreement connection >>>>>>>>>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, >>>>>>>>>>> >>>>>>>>>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> tree,cn=config), error 105 >>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>> cleanAllRUV_task: replica >>>>>>>>>>> (cn=meTovm-055.idm.lab. >>>>>>>>>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> tree, cn=config) has not been cleaned. You will need to rerun >>>>>>>>>>> the >>>>>>>>>>> CLEANALLRUV task on this replica. >>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>> cleanAllRUV_task: Task >>>>>>>>>>> failed (1) >>>>>>>>>>> >>>>>>>>>>> In this case I think we should inform user that the command failed, >>>>>>>>>>> possibly >>>>>>>>>>> because of disconnected replicas and that they could enable the >>>>>>>>>>> replicas and >>>>>>>>>>> try again. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 7) (minor) "pass" is now redundant in replication.py: >>>>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>>>>> + # We can't make the server we're removing read-only >>>>>>>>>>> but >>>>>>>>>>> + # this isn't a show-stopper >>>>>>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>>>>>> read-only, >>>>>>>>>>> continuing anyway") >>>>>>>>>>> + pass >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I think this addresses everything. >>>>>>>>>> >>>>>>>>>> rob >>>>>>>>> >>>>>>>>> Thanks, almost there! I just found one more issue which needs to be >>>>>>>>> fixed >>>>>>>>> before we push: >>>>>>>>> >>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>>>>>>> Directory Manager password: >>>>>>>>> >>>>>>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing >>>>>>>>> removal >>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>>> "Can't >>>>>>>>> contact LDAP server"} >>>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>>> >>>>>>>>> There were issues removing a connection: %d format: a number is >>>>>>>>> required, not str >>>>>>>>> >>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>>> "Can't >>>>>>>>> contact LDAP server"} >>>>>>>>> >>>>>>>>> This is a traceback I retrieved: >>>>>>>>> Traceback (most recent call last): >>>>>>>>> File "/sbin/ipa-replica-manage", line 425, in del_master >>>>>>>>> del_link(realm, r, hostname, options.dirman_passwd, force=True) >>>>>>>>> File "/sbin/ipa-replica-manage", line 271, in del_link >>>>>>>>> repl1.cleanallruv(replica_id) >>>>>>>>> File >>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>>>>>> line 1094, in cleanallruv >>>>>>>>> root_logger.debug("Creating CLEANALLRUV task for replica id >>>>>>>>> %d" % >>>>>>>>> replicaId) >>>>>>>>> >>>>>>>>> >>>>>>>>> The problem here is that you don't convert replica_id to int in this >>>>>>>>> part: >>>>>>>>> + replica_id = None >>>>>>>>> + if repl2: >>>>>>>>> + replica_id = repl2._get_replica_id(repl2.conn, None) >>>>>>>>> + else: >>>>>>>>> + servers = get_ruv(realm, replica1, dirman_passwd) >>>>>>>>> + for (netloc, rid) in servers: >>>>>>>>> + if netloc.startswith(replica2): >>>>>>>>> + replica_id = rid >>>>>>>>> + break >>>>>>>>> >>>>>>>>> Martin >>>>>>>>> >>>>>>>> >>>>>>>> Updated patch using new mechanism in 389-ds-base. This should more >>>>>>>> thoroughly clean out RUV data when a replica is being deleted, and >>>>>>>> provide for a way to delete RUV data afterwards too if necessary. >>>>>>>> >>>>>>>> rob >>>>>>> >>>>>>> Rebased patch >>>>>>> >>>>>>> rob >>>>>>> >>>>>> >>>>>> 0) As I wrote in a review for your patch 1041, changelog entry slipped >>>>>> elsewhere. >>>>>> >>>>>> 1) The following KeyboardInterrupt except class looks suspicious. I >>>>>> know why >>>>>> you have it there, but since it is generally a bad thing to do, some >>>>>> comment >>>>>> why it is needed would be useful. >>>>>> >>>>>> @@ -256,6 +263,17 @@ def del_link(realm, replica1, replica2, >>>>>> dirman_passwd, >>>>>> force=False): >>>>>> repl1.delete_agreement(replica2) >>>>>> repl1.delete_referral(replica2) >>>>>> >>>>>> + if type1 == replication.IPA_REPLICA: >>>>>> + if repl2: >>>>>> + ruv = repl2._get_replica_id(repl2.conn, None) >>>>>> + else: >>>>>> + ruv = get_ruv_by_host(realm, replica1, replica2, >>>>>> dirman_passwd) >>>>>> + >>>>>> + try: >>>>>> + repl1.cleanallruv(ruv) >>>>>> + except KeyboardInterrupt: >>>>>> + pass >>>>>> + >>>>>> >>>>>> Maybe you just wanted to do some cleanup and then "raise" again? >>>>> >>>>> No, it is there because it is safe to break out of it. The task will >>>>> continue to run. I added some verbiage. >>>>> >>>>>> >>>>>> 2) This is related to 1), but when some replica is down, >>>>>> "ipa-replica-manage >>>>>> del" may wait indefinitely when some remote replica is down, right? >>>>>> >>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>> Deleting a master is irreversible. >>>>>> To reconnect to the remote master you will need to prepare a new >>>>>> replica file >>>>>> and re-install. >>>>>> Continue to delete? [no]: y >>>>>> ipa: INFO: Setting agreement >>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>> >>>>>> >>>>>> >>>>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>> >>>>>> >>>>>> >>>>>> tree,cn=config >>>>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica >>>>>> acquired >>>>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>>>> Background task created to clean replication data >>>>>> >>>>>> ... after about a minute I hit CTRL+C >>>>>> >>>>>> ^CDeleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to >>>>>> 'vm-055.idm.lab.bos.redhat.com' >>>>>> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: NS record >>>>>> does not >>>>>> contain 'vm-055.idm.lab.bos.redhat.com.' >>>>>> You may need to manually remove them from the tree >>>>>> >>>>>> I think it would be better to inform user that some remote replica is >>>>>> down or >>>>>> at least that we are waiting for the task to complete. Something like >>>>>> that: >>>>>> >>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>> ... >>>>>> Background task created to clean replication data >>>>>> Replication data clean up may take very long time if some replica is >>>>>> unreachable >>>>>> Hit CTRL+C to interrupt the wait >>>>>> ^C Clean up wait interrupted >>>>>> .... >>>>>> [continue with del] >>>>> >>>>> Yup, did this in #1. >>>>> >>>>>> >>>>>> 3) (minor) When there is a cleanruv task running and you run >>>>>> "ipa-replica-manage del", there is a unexpected error message with >>>>>> duplicate >>>>>> task object in LDAP: >>>>>> >>>>>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com --force >>>>>> Unable to connect to replica vm-072.idm.lab.bos.redhat.com, forcing >>>>>> removal >>>>>> FAIL >>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>>> contact LDAP server"} >>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>> >>>>>> There were issues removing a connection: This entry already exists >>>>>> <<<<<<<<< >>>>>> >>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>>> contact LDAP server"} >>>>>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record >>>>>> does not >>>>>> contain 'vm-072.idm.lab.bos.redhat.com.' >>>>>> You may need to manually remove them from the tree >>>>>> >>>>>> >>>>>> I think it should be enough to just catch for "entry already exists" in >>>>>> cleanallruv function, and in such case print a relevant error message >>>>>> bail out. >>>>>> Thus, self.conn.checkTask(dn, dowait=True) would not be called too. >>>>> >>>>> Good catch, fixed. >>>>> >>>>>> >>>>>> >>>>>> 4) (minor): In make_readonly function, there is a redundant "pass" >>>>>> statement: >>>>>> >>>>>> + def make_readonly(self): >>>>>> + """ >>>>>> + Make the current replication agreement read-only. >>>>>> + """ >>>>>> + dn = DN(('cn', 'userRoot'), ('cn', 'ldbm database'), >>>>>> + ('cn', 'plugins'), ('cn', 'config')) >>>>>> + >>>>>> + mod = [(ldap.MOD_REPLACE, 'nsslapd-readonly', 'on')] >>>>>> + try: >>>>>> + self.conn.modify_s(dn, mod) >>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>> + # We can't make the server we're removing read-only but >>>>>> + # this isn't a show-stopper >>>>>> + root_logger.debug("No permission to switch replica to >>>>>> read-only, >>>>>> continuing anyway") >>>>>> + pass <<<<<<<<<<<<<<< >>>>> >>>>> Yeah, this is one of my common mistakes. I put in a pass initially, then >>>>> add logging in front of it and forget to delete the pass. Its gone now. >>>>> >>>>>> >>>>>> >>>>>> 5) In clean_ruv, I think allowing a --force option to bypass the >>>>>> user_input >>>>>> would be helpful (at least for test automation): >>>>>> >>>>>> + if not ipautil.user_input("Continue to clean?", False): >>>>>> + sys.exit("Aborted") >>>>> >>>>> Yup, added. >>>>> >>>>> rob >>>> >>>> Slightly revised patch. I still had a window open with one unsaved change. >>>> >>>> rob >>>> >>> >>> Apparently there were two unsaved changes, one of which was lost. This adds in >>> the 'entry already exists' fix. >>> >>> rob >>> >> >> Just one last thing (otherwise the patch is OK) - I don't think this is what we >> want :-) >> >> # ipa-replica-manage clean-ruv 8 >> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 >> >> Cleaning the wrong replica ID will cause that server to no >> longer replicate so it may miss updates while the process >> is running. It would need to be re-initialized to maintain >> consistency. Be very careful. >> Continue to clean? [no]: y <<<<<< >> Aborted >> >> >> Nor this exception, (your are checking for wrong exception): >> >> # ipa-replica-manage clean-ruv 8 >> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 >> >> Cleaning the wrong replica ID will cause that server to no >> longer replicate so it may miss updates while the process >> is running. It would need to be re-initialized to maintain >> consistency. Be very careful. >> Continue to clean? [no]: >> unexpected error: This entry already exists >> >> This is the exception: >> >> Traceback (most recent call last): >> File "/sbin/ipa-replica-manage", line 651, in >> main() >> File "/sbin/ipa-replica-manage", line 648, in main >> clean_ruv(realm, args[1], options) >> File "/sbin/ipa-replica-manage", line 373, in clean_ruv >> thisrepl.cleanallruv(ruv) >> File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >> line 1136, in cleanallruv >> self.conn.addEntry(e) >> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 503, in >> addEntry >> self.__handle_errors(e, arg_desc=arg_desc) >> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 321, in >> __handle_errors >> raise errors.DuplicateEntry() >> ipalib.errors.DuplicateEntry: This entry already exists >> >> Martin >> > > > On another matter, I just noticed that CLEANRUV is not proceeding if I have a > winsync replica defined (and it is even up): > > # ipa-replica-manage list > dc.ad.test: winsync <<<<<<< > vm-072.idm.lab.bos.redhat.com: master > vm-086.idm.lab.bos.redhat.com: master > > [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Waiting > for all the replicas to receive all the deleted replica updates... > [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Failed > to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), will retry later. > [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all > replicas caught up, retrying in 10 seconds > [06/Sep/2012:11:59:20 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Failed > to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), will retry later. > [06/Sep/2012:11:59:20 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all > replicas caught up, retrying in 20 seconds > [06/Sep/2012:11:59:40 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Failed > to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), will retry later. > [06/Sep/2012:11:59:40 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all > replicas caught up, retrying in 40 seconds > > I don't think this is OK. Adding Rich to CC to follow on this one. > > Martin > And now the actual CC. Martin From pviktori at redhat.com Thu Sep 6 16:13:15 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Thu, 06 Sep 2012 18:13:15 +0200 Subject: [Freeipa-devel] [PATCH 0013] Remove user-unfriendly "u" character from error messages In-Reply-To: <50476317.20604@redhat.com> References: <50475150.4090905@redhat.com> <504756AF.1050803@redhat.com> <50476317.20604@redhat.com> Message-ID: <5048CB9B.9040200@redhat.com> On 09/05/2012 04:35 PM, Tomas Babej wrote: > On 09/05/2012 03:42 PM, Petr Viktorin wrote: >> On 09/05/2012 03:19 PM, Tomas Babej wrote: >>> Hi, >>> >>> User-unfriendly errors were caused by re-raising errors >>> from external python module netaddr. >>> >>> https://fedorahosted.org/freeipa/ticket/2588 >>> >>> Tomas >>> >>> >>> _______________________________________________ >>> Freeipa-devel mailing list >>> Freeipa-devel at redhat.com >>> https://www.redhat.com/mailman/listinfo/freeipa-devel >>> >> >> I don't agree with this approach. Raising another module's errors in >> our code is ugly, and will break if netaddr changes. The arguments to >> pass to the exceptions are undocumented (see >> http://packages.python.org/netaddr/api.html#custom-exceptions). The >> wording of error messages in libraries can usually change at any time, >> and is intended for developers, not end users. >> >> This should be either fixed upstream (unlikely, using the repr() of >> the argument is a sane thing to do at their side), or we should pass >> bytestrings to netaddr (a possible quick fix, not sure it it'll work), >> or, ideally, we should raise IPA's own errors. >> > Well, this particular fix wouldn't have broken anything, since it was > raising the same error that the except clause in which the raising > occured caught. However, I changed this to StandardError, since the > error message is extracted and packed into ValidationError during > further validation and therefore simple format message is suitable. I know this is a minor issue and unlikely to cause problems, but it still should be fixed properly. The patch assumes AddrFormatError takes only one argument, the message. In another case something like this might be a reasonable assumption, but having a prettier error message doesn't justify it. Taking free-form text from a library and fixing it up like this is also not maintainable. Again, it assumes too much about the library. I won't ack this approach. Please consult someone else if you think it really is the best way. Adding `addr = str(addr)` would work around the issue without indroducing assumptions about an external library. Some technical issues with your patch, in case my "ideology" is incompatible with the project: ValueError would be more appropriate than StandardError. We already raise it in similar situations in this method. There is a case where your fix doesn't work: CheckedIPAddress(u'percent%sign'). Please adjust the test in test_dns_plugin that checks the error message. -- Petr? From rmeggins at redhat.com Thu Sep 6 16:13:18 2012 From: rmeggins at redhat.com (Rich Megginson) Date: Thu, 06 Sep 2012 10:13:18 -0600 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <5048CAD0.2070508@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> <50479491.1060200@redhat.com> <50488DD0.1080507@redhat.com> <5048BEF8.5000401@redhat.com> <5048C46A.8020207@redhat.com> <5048C754.2030708@redhat.com> <5048C9BE.1050005@redhat.com> <5048CA9F.7060607@redhat.com> <5048CAD0.2070508@redhat.com> Message-ID: <5048CB9E.4090301@redhat.com> On 09/06/2012 10:09 AM, Martin Kosek wrote: > On 09/06/2012 06:09 PM, Martin Kosek wrote: >> On 09/06/2012 06:05 PM, Martin Kosek wrote: >>> On 09/06/2012 05:55 PM, Rob Crittenden wrote: >>>> Rob Crittenden wrote: >>>>> Rob Crittenden wrote: >>>>>> Martin Kosek wrote: >>>>>>> On 09/05/2012 08:06 PM, Rob Crittenden wrote: >>>>>>>> Rob Crittenden wrote: >>>>>>>>> Martin Kosek wrote: >>>>>>>>>> On 07/05/2012 08:39 PM, Rob Crittenden wrote: >>>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: >>>>>>>>>>>>> Deleting a replica can leave a replication vector (RUV) on the >>>>>>>>>>>>> other servers. >>>>>>>>>>>>> This can confuse things if the replica is re-added, and it also >>>>>>>>>>>>> causes the >>>>>>>>>>>>> server to calculate changes against a server that may no longer >>>>>>>>>>>>> exist. >>>>>>>>>>>>> >>>>>>>>>>>>> 389-ds-base provides a new task that self-propogates itself to all >>>>>>>>>>>>> available >>>>>>>>>>>>> replicas to clean this RUV data. >>>>>>>>>>>>> >>>>>>>>>>>>> This patch will create this task at deletion time to hopefully >>>>>>>>>>>>> clean things up. >>>>>>>>>>>>> >>>>>>>>>>>>> It isn't perfect. If any replica is down or unavailable at the >>>>>>>>>>>>> time >>>>>>>>>>>>> the >>>>>>>>>>>>> cleanruv task fires, and then comes back up, the old RUV data >>>>>>>>>>>>> may be >>>>>>>>>>>>> re-propogated around. >>>>>>>>>>>>> >>>>>>>>>>>>> To make things easier in this case I've added two new commands to >>>>>>>>>>>>> ipa-replica-manage. The first lists the replication ids of all the >>>>>>>>>>>>> servers we >>>>>>>>>>>>> have a RUV for. Using this you can call clean_ruv with the >>>>>>>>>>>>> replication id of a >>>>>>>>>>>>> server that no longer exists to try the cleanallruv step again. >>>>>>>>>>>>> >>>>>>>>>>>>> This is quite dangerous though. If you run cleanruv against a >>>>>>>>>>>>> replica id that >>>>>>>>>>>>> does exist it can cause a loss of data. I believe I've put in >>>>>>>>>>>>> enough scary >>>>>>>>>>>>> warnings about this. >>>>>>>>>>>>> >>>>>>>>>>>>> rob >>>>>>>>>>>>> >>>>>>>>>>>> Good work there, this should make cleaning RUVs much easier than >>>>>>>>>>>> with the >>>>>>>>>>>> previous version. >>>>>>>>>>>> >>>>>>>>>>>> This is what I found during review: >>>>>>>>>>>> >>>>>>>>>>>> 1) list_ruv and clean_ruv command help in man is quite lost. I >>>>>>>>>>>> think >>>>>>>>>>>> it would >>>>>>>>>>>> help if we for example have all info for commands indented. This >>>>>>>>>>>> way >>>>>>>>>>>> user could >>>>>>>>>>>> simply over-look the new commands in the man page. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2) I would rename new commands to clean-ruv and list-ruv to make >>>>>>>>>>>> them >>>>>>>>>>>> consistent with the rest of the commands (re-initialize, >>>>>>>>>>>> force-sync). >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 3) It would be nice to be able to run clean_ruv command in an >>>>>>>>>>>> unattended way >>>>>>>>>>>> (for better testing), i.e. respect --force option as we already >>>>>>>>>>>> do for >>>>>>>>>>>> ipa-replica-manage del. This fix would aid test automation in the >>>>>>>>>>>> future. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 4) (minor) The new question (and the del too) does not react too >>>>>>>>>>>> well for >>>>>>>>>>>> CTRL+D: >>>>>>>>>>>> >>>>>>>>>>>> # ipa-replica-manage clean_ruv 3 --force >>>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>>> >>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>>> Continue to clean? [no]: unexpected error: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 5) Help for clean_ruv command without a required parameter is quite >>>>>>>>>>>> confusing >>>>>>>>>>>> as it reports that command is wrong and not the parameter: >>>>>>>>>>>> >>>>>>>>>>>> # ipa-replica-manage clean_ruv >>>>>>>>>>>> Usage: ipa-replica-manage [options] >>>>>>>>>>>> >>>>>>>>>>>> ipa-replica-manage: error: must provide a command [clean_ruv | >>>>>>>>>>>> force-sync | >>>>>>>>>>>> disconnect | connect | del | re-initialize | list | list_ruv] >>>>>>>>>>>> >>>>>>>>>>>> It seems you just forgot to specify the error message in the >>>>>>>>>>>> command >>>>>>>>>>>> definition >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 6) When the remote replica is down, the clean_ruv command fails >>>>>>>>>>>> with an >>>>>>>>>>>> unexpected error: >>>>>>>>>>>> >>>>>>>>>>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 >>>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>>> >>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>>> Continue to clean? [no]: y >>>>>>>>>>>> unexpected error: {'desc': 'Operations error'} >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: >>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>> cleanAllRUV_task: failed >>>>>>>>>>>> to connect to repl agreement connection >>>>>>>>>>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, >>>>>>>>>>>> >>>>>>>>>>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> tree,cn=config), error 105 >>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>> cleanAllRUV_task: replica >>>>>>>>>>>> (cn=meTovm-055.idm.lab. >>>>>>>>>>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> tree, cn=config) has not been cleaned. You will need to rerun >>>>>>>>>>>> the >>>>>>>>>>>> CLEANALLRUV task on this replica. >>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>> cleanAllRUV_task: Task >>>>>>>>>>>> failed (1) >>>>>>>>>>>> >>>>>>>>>>>> In this case I think we should inform user that the command failed, >>>>>>>>>>>> possibly >>>>>>>>>>>> because of disconnected replicas and that they could enable the >>>>>>>>>>>> replicas and >>>>>>>>>>>> try again. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 7) (minor) "pass" is now redundant in replication.py: >>>>>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>>>>>> + # We can't make the server we're removing read-only >>>>>>>>>>>> but >>>>>>>>>>>> + # this isn't a show-stopper >>>>>>>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>>>>>>> read-only, >>>>>>>>>>>> continuing anyway") >>>>>>>>>>>> + pass >>>>>>>>>>>> >>>>>>>>>>> I think this addresses everything. >>>>>>>>>>> >>>>>>>>>>> rob >>>>>>>>>> Thanks, almost there! I just found one more issue which needs to be >>>>>>>>>> fixed >>>>>>>>>> before we push: >>>>>>>>>> >>>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>>>>>>>> Directory Manager password: >>>>>>>>>> >>>>>>>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing >>>>>>>>>> removal >>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>>>> "Can't >>>>>>>>>> contact LDAP server"} >>>>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>>>> >>>>>>>>>> There were issues removing a connection: %d format: a number is >>>>>>>>>> required, not str >>>>>>>>>> >>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>>>> "Can't >>>>>>>>>> contact LDAP server"} >>>>>>>>>> >>>>>>>>>> This is a traceback I retrieved: >>>>>>>>>> Traceback (most recent call last): >>>>>>>>>> File "/sbin/ipa-replica-manage", line 425, in del_master >>>>>>>>>> del_link(realm, r, hostname, options.dirman_passwd, force=True) >>>>>>>>>> File "/sbin/ipa-replica-manage", line 271, in del_link >>>>>>>>>> repl1.cleanallruv(replica_id) >>>>>>>>>> File >>>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>>>>>>> line 1094, in cleanallruv >>>>>>>>>> root_logger.debug("Creating CLEANALLRUV task for replica id >>>>>>>>>> %d" % >>>>>>>>>> replicaId) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The problem here is that you don't convert replica_id to int in this >>>>>>>>>> part: >>>>>>>>>> + replica_id = None >>>>>>>>>> + if repl2: >>>>>>>>>> + replica_id = repl2._get_replica_id(repl2.conn, None) >>>>>>>>>> + else: >>>>>>>>>> + servers = get_ruv(realm, replica1, dirman_passwd) >>>>>>>>>> + for (netloc, rid) in servers: >>>>>>>>>> + if netloc.startswith(replica2): >>>>>>>>>> + replica_id = rid >>>>>>>>>> + break >>>>>>>>>> >>>>>>>>>> Martin >>>>>>>>>> >>>>>>>>> Updated patch using new mechanism in 389-ds-base. This should more >>>>>>>>> thoroughly clean out RUV data when a replica is being deleted, and >>>>>>>>> provide for a way to delete RUV data afterwards too if necessary. >>>>>>>>> >>>>>>>>> rob >>>>>>>> Rebased patch >>>>>>>> >>>>>>>> rob >>>>>>>> >>>>>>> 0) As I wrote in a review for your patch 1041, changelog entry slipped >>>>>>> elsewhere. >>>>>>> >>>>>>> 1) The following KeyboardInterrupt except class looks suspicious. I >>>>>>> know why >>>>>>> you have it there, but since it is generally a bad thing to do, some >>>>>>> comment >>>>>>> why it is needed would be useful. >>>>>>> >>>>>>> @@ -256,6 +263,17 @@ def del_link(realm, replica1, replica2, >>>>>>> dirman_passwd, >>>>>>> force=False): >>>>>>> repl1.delete_agreement(replica2) >>>>>>> repl1.delete_referral(replica2) >>>>>>> >>>>>>> + if type1 == replication.IPA_REPLICA: >>>>>>> + if repl2: >>>>>>> + ruv = repl2._get_replica_id(repl2.conn, None) >>>>>>> + else: >>>>>>> + ruv = get_ruv_by_host(realm, replica1, replica2, >>>>>>> dirman_passwd) >>>>>>> + >>>>>>> + try: >>>>>>> + repl1.cleanallruv(ruv) >>>>>>> + except KeyboardInterrupt: >>>>>>> + pass >>>>>>> + >>>>>>> >>>>>>> Maybe you just wanted to do some cleanup and then "raise" again? >>>>>> No, it is there because it is safe to break out of it. The task will >>>>>> continue to run. I added some verbiage. >>>>>> >>>>>>> 2) This is related to 1), but when some replica is down, >>>>>>> "ipa-replica-manage >>>>>>> del" may wait indefinitely when some remote replica is down, right? >>>>>>> >>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>>> Deleting a master is irreversible. >>>>>>> To reconnect to the remote master you will need to prepare a new >>>>>>> replica file >>>>>>> and re-install. >>>>>>> Continue to delete? [no]: y >>>>>>> ipa: INFO: Setting agreement >>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>>> >>>>>>> >>>>>>> >>>>>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>>>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>>> >>>>>>> >>>>>>> >>>>>>> tree,cn=config >>>>>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica >>>>>>> acquired >>>>>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>>>>> Background task created to clean replication data >>>>>>> >>>>>>> ... after about a minute I hit CTRL+C >>>>>>> >>>>>>> ^CDeleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to >>>>>>> 'vm-055.idm.lab.bos.redhat.com' >>>>>>> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: NS record >>>>>>> does not >>>>>>> contain 'vm-055.idm.lab.bos.redhat.com.' >>>>>>> You may need to manually remove them from the tree >>>>>>> >>>>>>> I think it would be better to inform user that some remote replica is >>>>>>> down or >>>>>>> at least that we are waiting for the task to complete. Something like >>>>>>> that: >>>>>>> >>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>>> ... >>>>>>> Background task created to clean replication data >>>>>>> Replication data clean up may take very long time if some replica is >>>>>>> unreachable >>>>>>> Hit CTRL+C to interrupt the wait >>>>>>> ^C Clean up wait interrupted >>>>>>> .... >>>>>>> [continue with del] >>>>>> Yup, did this in #1. >>>>>> >>>>>>> 3) (minor) When there is a cleanruv task running and you run >>>>>>> "ipa-replica-manage del", there is a unexpected error message with >>>>>>> duplicate >>>>>>> task object in LDAP: >>>>>>> >>>>>>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com --force >>>>>>> Unable to connect to replica vm-072.idm.lab.bos.redhat.com, forcing >>>>>>> removal >>>>>>> FAIL >>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>>>> contact LDAP server"} >>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>> >>>>>>> There were issues removing a connection: This entry already exists >>>>>>> <<<<<<<<< >>>>>>> >>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>>>> contact LDAP server"} >>>>>>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record >>>>>>> does not >>>>>>> contain 'vm-072.idm.lab.bos.redhat.com.' >>>>>>> You may need to manually remove them from the tree >>>>>>> >>>>>>> >>>>>>> I think it should be enough to just catch for "entry already exists" in >>>>>>> cleanallruv function, and in such case print a relevant error message >>>>>>> bail out. >>>>>>> Thus, self.conn.checkTask(dn, dowait=True) would not be called too. >>>>>> Good catch, fixed. >>>>>> >>>>>>> >>>>>>> 4) (minor): In make_readonly function, there is a redundant "pass" >>>>>>> statement: >>>>>>> >>>>>>> + def make_readonly(self): >>>>>>> + """ >>>>>>> + Make the current replication agreement read-only. >>>>>>> + """ >>>>>>> + dn = DN(('cn', 'userRoot'), ('cn', 'ldbm database'), >>>>>>> + ('cn', 'plugins'), ('cn', 'config')) >>>>>>> + >>>>>>> + mod = [(ldap.MOD_REPLACE, 'nsslapd-readonly', 'on')] >>>>>>> + try: >>>>>>> + self.conn.modify_s(dn, mod) >>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>> + # We can't make the server we're removing read-only but >>>>>>> + # this isn't a show-stopper >>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>> read-only, >>>>>>> continuing anyway") >>>>>>> + pass<<<<<<<<<<<<<<< >>>>>> Yeah, this is one of my common mistakes. I put in a pass initially, then >>>>>> add logging in front of it and forget to delete the pass. Its gone now. >>>>>> >>>>>>> >>>>>>> 5) In clean_ruv, I think allowing a --force option to bypass the >>>>>>> user_input >>>>>>> would be helpful (at least for test automation): >>>>>>> >>>>>>> + if not ipautil.user_input("Continue to clean?", False): >>>>>>> + sys.exit("Aborted") >>>>>> Yup, added. >>>>>> >>>>>> rob >>>>> Slightly revised patch. I still had a window open with one unsaved change. >>>>> >>>>> rob >>>>> >>>> Apparently there were two unsaved changes, one of which was lost. This adds in >>>> the 'entry already exists' fix. >>>> >>>> rob >>>> >>> Just one last thing (otherwise the patch is OK) - I don't think this is what we >>> want :-) >>> >>> # ipa-replica-manage clean-ruv 8 >>> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 >>> >>> Cleaning the wrong replica ID will cause that server to no >>> longer replicate so it may miss updates while the process >>> is running. It would need to be re-initialized to maintain >>> consistency. Be very careful. >>> Continue to clean? [no]: y<<<<<< >>> Aborted >>> >>> >>> Nor this exception, (your are checking for wrong exception): >>> >>> # ipa-replica-manage clean-ruv 8 >>> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 >>> >>> Cleaning the wrong replica ID will cause that server to no >>> longer replicate so it may miss updates while the process >>> is running. It would need to be re-initialized to maintain >>> consistency. Be very careful. >>> Continue to clean? [no]: >>> unexpected error: This entry already exists >>> >>> This is the exception: >>> >>> Traceback (most recent call last): >>> File "/sbin/ipa-replica-manage", line 651, in >>> main() >>> File "/sbin/ipa-replica-manage", line 648, in main >>> clean_ruv(realm, args[1], options) >>> File "/sbin/ipa-replica-manage", line 373, in clean_ruv >>> thisrepl.cleanallruv(ruv) >>> File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>> line 1136, in cleanallruv >>> self.conn.addEntry(e) >>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 503, in >>> addEntry >>> self.__handle_errors(e, arg_desc=arg_desc) >>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 321, in >>> __handle_errors >>> raise errors.DuplicateEntry() >>> ipalib.errors.DuplicateEntry: This entry already exists >>> >>> Martin >>> >> >> On another matter, I just noticed that CLEANRUV is not proceeding if I have a >> winsync replica defined (and it is even up): >> >> # ipa-replica-manage list >> dc.ad.test: winsync<<<<<<< >> vm-072.idm.lab.bos.redhat.com: master >> vm-086.idm.lab.bos.redhat.com: master >> >> [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Waiting >> for all the replicas to receive all the deleted replica updates... >> [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Failed >> to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), will retry later. >> [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all >> replicas caught up, retrying in 10 seconds >> [06/Sep/2012:11:59:20 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Failed >> to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), will retry later. >> [06/Sep/2012:11:59:20 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all >> replicas caught up, retrying in 20 seconds >> [06/Sep/2012:11:59:40 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Failed >> to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), will retry later. >> [06/Sep/2012:11:59:40 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all >> replicas caught up, retrying in 40 seconds >> >> I don't think this is OK. Adding Rich to CC to follow on this one. >> >> Martin >> > And now the actual CC. > Martin > Yeah - looks like CLEANALLRUV needs to ignore windows sync agreements From mkosek at redhat.com Thu Sep 6 16:27:17 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 06 Sep 2012 18:27:17 +0200 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <5048CB9E.4090301@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> <50479491.1060200@redhat.com> <50488DD0.1080507@redhat.com> <5048BEF8.5000401@redhat.com> <5048C46A.8020207@redhat.com> <5048C754.2030708@redhat.com> <5048C9BE.1050005@redhat.com> <5048CA9F.7060607@redhat.com> <5048CAD0.2070508@redhat.com> <5048CB9E.4090301@redhat.com> Message-ID: <5048CEE5.80704@redhat.com> On 09/06/2012 06:13 PM, Rich Megginson wrote: > On 09/06/2012 10:09 AM, Martin Kosek wrote: >> On 09/06/2012 06:09 PM, Martin Kosek wrote: >>> On 09/06/2012 06:05 PM, Martin Kosek wrote: >>>> On 09/06/2012 05:55 PM, Rob Crittenden wrote: >>>>> Rob Crittenden wrote: >>>>>> Rob Crittenden wrote: >>>>>>> Martin Kosek wrote: >>>>>>>> On 09/05/2012 08:06 PM, Rob Crittenden wrote: >>>>>>>>> Rob Crittenden wrote: >>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>> On 07/05/2012 08:39 PM, Rob Crittenden wrote: >>>>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: >>>>>>>>>>>>>> Deleting a replica can leave a replication vector (RUV) on the >>>>>>>>>>>>>> other servers. >>>>>>>>>>>>>> This can confuse things if the replica is re-added, and it also >>>>>>>>>>>>>> causes the >>>>>>>>>>>>>> server to calculate changes against a server that may no longer >>>>>>>>>>>>>> exist. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 389-ds-base provides a new task that self-propogates itself to all >>>>>>>>>>>>>> available >>>>>>>>>>>>>> replicas to clean this RUV data. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This patch will create this task at deletion time to hopefully >>>>>>>>>>>>>> clean things up. >>>>>>>>>>>>>> >>>>>>>>>>>>>> It isn't perfect. If any replica is down or unavailable at the >>>>>>>>>>>>>> time >>>>>>>>>>>>>> the >>>>>>>>>>>>>> cleanruv task fires, and then comes back up, the old RUV data >>>>>>>>>>>>>> may be >>>>>>>>>>>>>> re-propogated around. >>>>>>>>>>>>>> >>>>>>>>>>>>>> To make things easier in this case I've added two new commands to >>>>>>>>>>>>>> ipa-replica-manage. The first lists the replication ids of all the >>>>>>>>>>>>>> servers we >>>>>>>>>>>>>> have a RUV for. Using this you can call clean_ruv with the >>>>>>>>>>>>>> replication id of a >>>>>>>>>>>>>> server that no longer exists to try the cleanallruv step again. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is quite dangerous though. If you run cleanruv against a >>>>>>>>>>>>>> replica id that >>>>>>>>>>>>>> does exist it can cause a loss of data. I believe I've put in >>>>>>>>>>>>>> enough scary >>>>>>>>>>>>>> warnings about this. >>>>>>>>>>>>>> >>>>>>>>>>>>>> rob >>>>>>>>>>>>>> >>>>>>>>>>>>> Good work there, this should make cleaning RUVs much easier than >>>>>>>>>>>>> with the >>>>>>>>>>>>> previous version. >>>>>>>>>>>>> >>>>>>>>>>>>> This is what I found during review: >>>>>>>>>>>>> >>>>>>>>>>>>> 1) list_ruv and clean_ruv command help in man is quite lost. I >>>>>>>>>>>>> think >>>>>>>>>>>>> it would >>>>>>>>>>>>> help if we for example have all info for commands indented. This >>>>>>>>>>>>> way >>>>>>>>>>>>> user could >>>>>>>>>>>>> simply over-look the new commands in the man page. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 2) I would rename new commands to clean-ruv and list-ruv to make >>>>>>>>>>>>> them >>>>>>>>>>>>> consistent with the rest of the commands (re-initialize, >>>>>>>>>>>>> force-sync). >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 3) It would be nice to be able to run clean_ruv command in an >>>>>>>>>>>>> unattended way >>>>>>>>>>>>> (for better testing), i.e. respect --force option as we already >>>>>>>>>>>>> do for >>>>>>>>>>>>> ipa-replica-manage del. This fix would aid test automation in the >>>>>>>>>>>>> future. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 4) (minor) The new question (and the del too) does not react too >>>>>>>>>>>>> well for >>>>>>>>>>>>> CTRL+D: >>>>>>>>>>>>> >>>>>>>>>>>>> # ipa-replica-manage clean_ruv 3 --force >>>>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>>>> >>>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>>>> Continue to clean? [no]: unexpected error: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 5) Help for clean_ruv command without a required parameter is quite >>>>>>>>>>>>> confusing >>>>>>>>>>>>> as it reports that command is wrong and not the parameter: >>>>>>>>>>>>> >>>>>>>>>>>>> # ipa-replica-manage clean_ruv >>>>>>>>>>>>> Usage: ipa-replica-manage [options] >>>>>>>>>>>>> >>>>>>>>>>>>> ipa-replica-manage: error: must provide a command [clean_ruv | >>>>>>>>>>>>> force-sync | >>>>>>>>>>>>> disconnect | connect | del | re-initialize | list | list_ruv] >>>>>>>>>>>>> >>>>>>>>>>>>> It seems you just forgot to specify the error message in the >>>>>>>>>>>>> command >>>>>>>>>>>>> definition >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 6) When the remote replica is down, the clean_ruv command fails >>>>>>>>>>>>> with an >>>>>>>>>>>>> unexpected error: >>>>>>>>>>>>> >>>>>>>>>>>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 >>>>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>>>> >>>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>>>> Continue to clean? [no]: y >>>>>>>>>>>>> unexpected error: {'desc': 'Operations error'} >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: >>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>> cleanAllRUV_task: failed >>>>>>>>>>>>> to connect to repl agreement connection >>>>>>>>>>>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, >>>>>>>>>>>>> >>>>>>>>>>>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> tree,cn=config), error 105 >>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>> cleanAllRUV_task: replica >>>>>>>>>>>>> (cn=meTovm-055.idm.lab. >>>>>>>>>>>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> tree, cn=config) has not been cleaned. You will need to rerun >>>>>>>>>>>>> the >>>>>>>>>>>>> CLEANALLRUV task on this replica. >>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>> cleanAllRUV_task: Task >>>>>>>>>>>>> failed (1) >>>>>>>>>>>>> >>>>>>>>>>>>> In this case I think we should inform user that the command failed, >>>>>>>>>>>>> possibly >>>>>>>>>>>>> because of disconnected replicas and that they could enable the >>>>>>>>>>>>> replicas and >>>>>>>>>>>>> try again. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 7) (minor) "pass" is now redundant in replication.py: >>>>>>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>>>>>>> + # We can't make the server we're removing read-only >>>>>>>>>>>>> but >>>>>>>>>>>>> + # this isn't a show-stopper >>>>>>>>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>>>>>>>> read-only, >>>>>>>>>>>>> continuing anyway") >>>>>>>>>>>>> + pass >>>>>>>>>>>>> >>>>>>>>>>>> I think this addresses everything. >>>>>>>>>>>> >>>>>>>>>>>> rob >>>>>>>>>>> Thanks, almost there! I just found one more issue which needs to be >>>>>>>>>>> fixed >>>>>>>>>>> before we push: >>>>>>>>>>> >>>>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>>>>>>>>> Directory Manager password: >>>>>>>>>>> >>>>>>>>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing >>>>>>>>>>> removal >>>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>>>>> "Can't >>>>>>>>>>> contact LDAP server"} >>>>>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>>>>> >>>>>>>>>>> There were issues removing a connection: %d format: a number is >>>>>>>>>>> required, not str >>>>>>>>>>> >>>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>>>>> "Can't >>>>>>>>>>> contact LDAP server"} >>>>>>>>>>> >>>>>>>>>>> This is a traceback I retrieved: >>>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>> File "/sbin/ipa-replica-manage", line 425, in del_master >>>>>>>>>>> del_link(realm, r, hostname, options.dirman_passwd, force=True) >>>>>>>>>>> File "/sbin/ipa-replica-manage", line 271, in del_link >>>>>>>>>>> repl1.cleanallruv(replica_id) >>>>>>>>>>> File >>>>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>>>>>>>> line 1094, in cleanallruv >>>>>>>>>>> root_logger.debug("Creating CLEANALLRUV task for replica id >>>>>>>>>>> %d" % >>>>>>>>>>> replicaId) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> The problem here is that you don't convert replica_id to int in this >>>>>>>>>>> part: >>>>>>>>>>> + replica_id = None >>>>>>>>>>> + if repl2: >>>>>>>>>>> + replica_id = repl2._get_replica_id(repl2.conn, None) >>>>>>>>>>> + else: >>>>>>>>>>> + servers = get_ruv(realm, replica1, dirman_passwd) >>>>>>>>>>> + for (netloc, rid) in servers: >>>>>>>>>>> + if netloc.startswith(replica2): >>>>>>>>>>> + replica_id = rid >>>>>>>>>>> + break >>>>>>>>>>> >>>>>>>>>>> Martin >>>>>>>>>>> >>>>>>>>>> Updated patch using new mechanism in 389-ds-base. This should more >>>>>>>>>> thoroughly clean out RUV data when a replica is being deleted, and >>>>>>>>>> provide for a way to delete RUV data afterwards too if necessary. >>>>>>>>>> >>>>>>>>>> rob >>>>>>>>> Rebased patch >>>>>>>>> >>>>>>>>> rob >>>>>>>>> >>>>>>>> 0) As I wrote in a review for your patch 1041, changelog entry slipped >>>>>>>> elsewhere. >>>>>>>> >>>>>>>> 1) The following KeyboardInterrupt except class looks suspicious. I >>>>>>>> know why >>>>>>>> you have it there, but since it is generally a bad thing to do, some >>>>>>>> comment >>>>>>>> why it is needed would be useful. >>>>>>>> >>>>>>>> @@ -256,6 +263,17 @@ def del_link(realm, replica1, replica2, >>>>>>>> dirman_passwd, >>>>>>>> force=False): >>>>>>>> repl1.delete_agreement(replica2) >>>>>>>> repl1.delete_referral(replica2) >>>>>>>> >>>>>>>> + if type1 == replication.IPA_REPLICA: >>>>>>>> + if repl2: >>>>>>>> + ruv = repl2._get_replica_id(repl2.conn, None) >>>>>>>> + else: >>>>>>>> + ruv = get_ruv_by_host(realm, replica1, replica2, >>>>>>>> dirman_passwd) >>>>>>>> + >>>>>>>> + try: >>>>>>>> + repl1.cleanallruv(ruv) >>>>>>>> + except KeyboardInterrupt: >>>>>>>> + pass >>>>>>>> + >>>>>>>> >>>>>>>> Maybe you just wanted to do some cleanup and then "raise" again? >>>>>>> No, it is there because it is safe to break out of it. The task will >>>>>>> continue to run. I added some verbiage. >>>>>>> >>>>>>>> 2) This is related to 1), but when some replica is down, >>>>>>>> "ipa-replica-manage >>>>>>>> del" may wait indefinitely when some remote replica is down, right? >>>>>>>> >>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>>>> Deleting a master is irreversible. >>>>>>>> To reconnect to the remote master you will need to prepare a new >>>>>>>> replica file >>>>>>>> and re-install. >>>>>>>> Continue to delete? [no]: y >>>>>>>> ipa: INFO: Setting agreement >>>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>>>>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> tree,cn=config >>>>>>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica >>>>>>>> acquired >>>>>>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>>>>>> Background task created to clean replication data >>>>>>>> >>>>>>>> ... after about a minute I hit CTRL+C >>>>>>>> >>>>>>>> ^CDeleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to >>>>>>>> 'vm-055.idm.lab.bos.redhat.com' >>>>>>>> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: NS record >>>>>>>> does not >>>>>>>> contain 'vm-055.idm.lab.bos.redhat.com.' >>>>>>>> You may need to manually remove them from the tree >>>>>>>> >>>>>>>> I think it would be better to inform user that some remote replica is >>>>>>>> down or >>>>>>>> at least that we are waiting for the task to complete. Something like >>>>>>>> that: >>>>>>>> >>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>>>> ... >>>>>>>> Background task created to clean replication data >>>>>>>> Replication data clean up may take very long time if some replica is >>>>>>>> unreachable >>>>>>>> Hit CTRL+C to interrupt the wait >>>>>>>> ^C Clean up wait interrupted >>>>>>>> .... >>>>>>>> [continue with del] >>>>>>> Yup, did this in #1. >>>>>>> >>>>>>>> 3) (minor) When there is a cleanruv task running and you run >>>>>>>> "ipa-replica-manage del", there is a unexpected error message with >>>>>>>> duplicate >>>>>>>> task object in LDAP: >>>>>>>> >>>>>>>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com --force >>>>>>>> Unable to connect to replica vm-072.idm.lab.bos.redhat.com, forcing >>>>>>>> removal >>>>>>>> FAIL >>>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>>>>> contact LDAP server"} >>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>> >>>>>>>> There were issues removing a connection: This entry already exists >>>>>>>> <<<<<<<<< >>>>>>>> >>>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>>>>> contact LDAP server"} >>>>>>>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record >>>>>>>> does not >>>>>>>> contain 'vm-072.idm.lab.bos.redhat.com.' >>>>>>>> You may need to manually remove them from the tree >>>>>>>> >>>>>>>> >>>>>>>> I think it should be enough to just catch for "entry already exists" in >>>>>>>> cleanallruv function, and in such case print a relevant error message >>>>>>>> bail out. >>>>>>>> Thus, self.conn.checkTask(dn, dowait=True) would not be called too. >>>>>>> Good catch, fixed. >>>>>>> >>>>>>>> >>>>>>>> 4) (minor): In make_readonly function, there is a redundant "pass" >>>>>>>> statement: >>>>>>>> >>>>>>>> + def make_readonly(self): >>>>>>>> + """ >>>>>>>> + Make the current replication agreement read-only. >>>>>>>> + """ >>>>>>>> + dn = DN(('cn', 'userRoot'), ('cn', 'ldbm database'), >>>>>>>> + ('cn', 'plugins'), ('cn', 'config')) >>>>>>>> + >>>>>>>> + mod = [(ldap.MOD_REPLACE, 'nsslapd-readonly', 'on')] >>>>>>>> + try: >>>>>>>> + self.conn.modify_s(dn, mod) >>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>> + # We can't make the server we're removing read-only but >>>>>>>> + # this isn't a show-stopper >>>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>>> read-only, >>>>>>>> continuing anyway") >>>>>>>> + pass<<<<<<<<<<<<<<< >>>>>>> Yeah, this is one of my common mistakes. I put in a pass initially, then >>>>>>> add logging in front of it and forget to delete the pass. Its gone now. >>>>>>> >>>>>>>> >>>>>>>> 5) In clean_ruv, I think allowing a --force option to bypass the >>>>>>>> user_input >>>>>>>> would be helpful (at least for test automation): >>>>>>>> >>>>>>>> + if not ipautil.user_input("Continue to clean?", False): >>>>>>>> + sys.exit("Aborted") >>>>>>> Yup, added. >>>>>>> >>>>>>> rob >>>>>> Slightly revised patch. I still had a window open with one unsaved change. >>>>>> >>>>>> rob >>>>>> >>>>> Apparently there were two unsaved changes, one of which was lost. This >>>>> adds in >>>>> the 'entry already exists' fix. >>>>> >>>>> rob >>>>> >>>> Just one last thing (otherwise the patch is OK) - I don't think this is >>>> what we >>>> want :-) >>>> >>>> # ipa-replica-manage clean-ruv 8 >>>> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 >>>> >>>> Cleaning the wrong replica ID will cause that server to no >>>> longer replicate so it may miss updates while the process >>>> is running. It would need to be re-initialized to maintain >>>> consistency. Be very careful. >>>> Continue to clean? [no]: y<<<<<< >>>> Aborted >>>> >>>> >>>> Nor this exception, (your are checking for wrong exception): >>>> >>>> # ipa-replica-manage clean-ruv 8 >>>> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 >>>> >>>> Cleaning the wrong replica ID will cause that server to no >>>> longer replicate so it may miss updates while the process >>>> is running. It would need to be re-initialized to maintain >>>> consistency. Be very careful. >>>> Continue to clean? [no]: >>>> unexpected error: This entry already exists >>>> >>>> This is the exception: >>>> >>>> Traceback (most recent call last): >>>> File "/sbin/ipa-replica-manage", line 651, in >>>> main() >>>> File "/sbin/ipa-replica-manage", line 648, in main >>>> clean_ruv(realm, args[1], options) >>>> File "/sbin/ipa-replica-manage", line 373, in clean_ruv >>>> thisrepl.cleanallruv(ruv) >>>> File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>> line 1136, in cleanallruv >>>> self.conn.addEntry(e) >>>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 503, in >>>> addEntry >>>> self.__handle_errors(e, arg_desc=arg_desc) >>>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 321, in >>>> __handle_errors >>>> raise errors.DuplicateEntry() >>>> ipalib.errors.DuplicateEntry: This entry already exists >>>> >>>> Martin >>>> >>> >>> On another matter, I just noticed that CLEANRUV is not proceeding if I have a >>> winsync replica defined (and it is even up): >>> >>> # ipa-replica-manage list >>> dc.ad.test: winsync<<<<<<< >>> vm-072.idm.lab.bos.redhat.com: master >>> vm-086.idm.lab.bos.redhat.com: master >>> >>> [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Waiting >>> for all the replicas to receive all the deleted replica updates... >>> [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Failed >>> to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), will retry >>> later. >>> [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all >>> replicas caught up, retrying in 10 seconds >>> [06/Sep/2012:11:59:20 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Failed >>> to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), will retry >>> later. >>> [06/Sep/2012:11:59:20 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all >>> replicas caught up, retrying in 20 seconds >>> [06/Sep/2012:11:59:40 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Failed >>> to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), will retry >>> later. >>> [06/Sep/2012:11:59:40 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all >>> replicas caught up, retrying in 40 seconds >>> >>> I don't think this is OK. Adding Rich to CC to follow on this one. >>> >>> Martin >>> >> And now the actual CC. >> Martin >> > Yeah - looks like CLEANALLRUV needs to ignore windows sync agreements Have you created a 389 ticket for that or would you like me to do that? While talking about this issue, does DS also cover a situation when one of the replicas does not support CLEANRUV task? Would not CLEANRUV stumble similarly to winsync agreement? Martin From mkosek at redhat.com Thu Sep 6 16:28:45 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 06 Sep 2012 18:28:45 +0200 Subject: [Freeipa-devel] Ticket #2866 - referential integrity in IPA In-Reply-To: <504612DC.9060402@redhat.com> References: <1345211089.3796.34.camel@priserak> <502E4E78.1090407@redhat.com> <503B9F4E.4010005@redhat.com> <503BA065.1060207@redhat.com> <5044C1F0.7080609@redhat.com> <504612DC.9060402@redhat.com> Message-ID: <5048CF3D.8020408@redhat.com> On 09/04/2012 04:40 PM, Rich Megginson wrote: > On 09/03/2012 08:42 AM, Martin Kosek wrote: >> On 08/27/2012 06:29 PM, Rich Megginson wrote: >>> On 08/27/2012 10:24 AM, Martin Kosek wrote: >>>> On 08/17/2012 04:00 PM, Rich Megginson wrote: >>>>> On 08/17/2012 07:44 AM, Martin Kosek wrote: >>>>>> Hi guys, >>>>>> >>>>>> I am now investigating ticket #2866: >>>>>> https://fedorahosted.org/freeipa/ticket/2866 >>>>>> >>>>>> And I am thinking about possible solutions for this problem. In a >>>>>> nutshell, we do not properly check referential integrity in some IPA >>>>>> objects where we keep one-way DN references to other objects, e.g. in >>>>>> - managedBy attribute for a host object >>>>>> - memberhost attribute for HBAC rule object >>>>>> - memberuser attribute for user object >>>>>> - memberallowcmd or memberdenycmd for SUDO command object (reported in >>>>>> #2866) >>>>>> ... >>>>>> >>>>>> Currently, I see 2 approaches to solve this: >>>>>> 1) Add relevant checks to our ipalib plugins where problematic >>>>>> operations with these operations are being executed (like we do for >>>>>> selinuxusermap's seealso attribute in HBAC plugin) >>>>>> This of course would not prevent direct LDAP deletes. >>>>>> >>>>>> 2) Implement a preop DS plugin that would hook to MODRDN and DELETE >>>>>> callbacks and check that this object's DN is not referenced in other >>>>>> objects. And if it does, it would reject such modification. Second >>>>>> option would be to delete the attribute value with now invalid >>>>>> reference. This would be probably more suitable for example for >>>>>> references to user objects. >>>>>> >>>>>> Any comments to these possible approaches are welcome. >>>>>> >>>>>> Rich, do you think that as an alternative to these 2 approaches, >>>>>> memberOf plugin could be eventually modified to do this task? >>>>> This is very similar to the referential integrity plugin already in 389, >>>>> except >>>>> instead of cleaning up references to moved and deleted entries, you want >>>>> it to >>>>> prevent moving or deleting an entry if that entry is referenced by the >>>>> managedby/memberhost/memberuser/memberallowcmd/memberdenycmd of some other >>>>> entry. >>>> I think that using or enhancing current DS referential integrity plugin >>>> will be >>>> the best and the most consistent way to go. >>>> >>>> We already use that plugin for some user attributes like "manager" or >>>> "secretary". seeAlso is already covered by default, so for example seeAlso >>>> attribute in SELinux usermap object referencing an HBAC rule will get removed >>>> when relevant HBAC rule is removed (I just checked that). >>>> >>>>> Note that the managed entry plugin (mep) already handles this for the >>>>> managedby >>>>> attribute. >>>> I assume you are referencing "mepmanagedby" and "mepmanagedentry" attributes >>>> which then produce errors like this one: >>>> >>>> # ipa netgroup-del foo >>>> ipa: ERROR: Server is unwilling to perform: Deleting a managed entry is not >>>> allowed. It needs to be manually unlinked first. >>>> >>>> managedBy attribute used by hosts objects I had in mind seems to not be >>>> covered. >>>> >>>> But you are right, this is pretty much what I wanted. Though in case of MEP, >>>> there is a link in both referenced objects, but in our case, we have just the >>>> one-way link. >>>> >>>>> Are you already using the memberof plugin for >>>>> memberhost/memberuser/memberallowcmd/memberdenycmd? >>>>> >>>>> This doesn't seem like a job for memberof, this seems like more of a new >>>>> check >>>>> for the referential integrity plugin. >>>>> >>>> I am now considering if current move/delete clean up already present in >>>> Referential Integrity plugin would not be sufficient for us. >>>> >>>> Rich, please correct me if I am wrong, but in that case, we would just need to >>>> add relevant attribute names >>>> (memberhost/memberuser/memberallowcmd/memberdenycmd...) to Referential >>>> Integrity plugin configuration as nsslapd-pluginarg7, nsslapd-pluginarg8, ... >>>> I wonder if there would be some performance issues if we add attributes to the >>>> list this way. >>> No, not if they are indexed for presence and equality. >>> >> Hello Rich, >> I am back to investigate this ticket. In order to be able to deliver some >> working solution to IPA 3.0, I plan to take advantage of current Referential >> Integrity Plugin to clean up dangling references. >> >> This is the plan I plan to take: >> 1) Add pres,eq indexes for all un-indexed attributes that we want to check: >> sourcehost >> memberservice >> managedby >> memberallowcmd >> memberdenycmd >> ipasudorunas >> ipasudorunasgroup > ok >> >> 2) Add missing "pres" index for attributes we want to check, but only have eq >> index: >> manager >> secretary >> memberuser >> memberhost >> >> I assume this step is also needed in order to keep the server performance. > yes >> >> 3) Add all these attributes do Referential Integrity Plugin attribute list of >> not already > ok >> >> 4) Also add Index task (nsIndexAttribute) for all these new indexes so that >> they are created during IPA server upgrade. > ok >> >> Is this procedure OK DS-wise? > Yes >> >> I also have question regarding the following note in RHDS doc chapter 3.6. >> Maintaining Referential Integrity: >> >> "The Referential Integrity Plug-in should only be enabled on one supplier >> replica in a multi-master replication environment to avoid conflict resolution >> loops..." >> >> Currently, we enable this plugin on all IPA replicas. Is this something we need >> to be concerned about and fix ASAP (before we do all this RefInt effort)? > > Mark/Nathan - I know you guys have looked at this issue. Hello guys, any update on that? Just checking before I implement this extended referential check in IPA. Thanks, Martin From mareynol at redhat.com Thu Sep 6 16:40:35 2012 From: mareynol at redhat.com (Mark Reynolds) Date: Thu, 06 Sep 2012 12:40:35 -0400 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <5048CB9E.4090301@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> <50479491.1060200@redhat.com> <50488DD0.1080507@redhat.com> <5048BEF8.5000401@redhat.com> <5048C46A.8020207@redhat.com> <5048C754.2030708@redhat.com> <5048C9BE.1050005@redhat.com> <5048CA9F.7060607@redhat.com> <5048CAD0.2070508@redhat.com> <5048CB9E.4090301@redhat.com> Message-ID: <5048D203.30807@redhat.com> On 09/06/2012 12:13 PM, Rich Megginson wrote: > On 09/06/2012 10:09 AM, Martin Kosek wrote: >> On 09/06/2012 06:09 PM, Martin Kosek wrote: >>> On 09/06/2012 06:05 PM, Martin Kosek wrote: >>>> On 09/06/2012 05:55 PM, Rob Crittenden wrote: >>>>> Rob Crittenden wrote: >>>>>> Rob Crittenden wrote: >>>>>>> Martin Kosek wrote: >>>>>>>> On 09/05/2012 08:06 PM, Rob Crittenden wrote: >>>>>>>>> Rob Crittenden wrote: >>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>> On 07/05/2012 08:39 PM, Rob Crittenden wrote: >>>>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: >>>>>>>>>>>>>> Deleting a replica can leave a replication vector (RUV) >>>>>>>>>>>>>> on the >>>>>>>>>>>>>> other servers. >>>>>>>>>>>>>> This can confuse things if the replica is re-added, and >>>>>>>>>>>>>> it also >>>>>>>>>>>>>> causes the >>>>>>>>>>>>>> server to calculate changes against a server that may no >>>>>>>>>>>>>> longer >>>>>>>>>>>>>> exist. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 389-ds-base provides a new task that self-propogates >>>>>>>>>>>>>> itself to all >>>>>>>>>>>>>> available >>>>>>>>>>>>>> replicas to clean this RUV data. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This patch will create this task at deletion time to >>>>>>>>>>>>>> hopefully >>>>>>>>>>>>>> clean things up. >>>>>>>>>>>>>> >>>>>>>>>>>>>> It isn't perfect. If any replica is down or unavailable >>>>>>>>>>>>>> at the >>>>>>>>>>>>>> time >>>>>>>>>>>>>> the >>>>>>>>>>>>>> cleanruv task fires, and then comes back up, the old RUV >>>>>>>>>>>>>> data >>>>>>>>>>>>>> may be >>>>>>>>>>>>>> re-propogated around. >>>>>>>>>>>>>> >>>>>>>>>>>>>> To make things easier in this case I've added two new >>>>>>>>>>>>>> commands to >>>>>>>>>>>>>> ipa-replica-manage. The first lists the replication ids >>>>>>>>>>>>>> of all the >>>>>>>>>>>>>> servers we >>>>>>>>>>>>>> have a RUV for. Using this you can call clean_ruv with the >>>>>>>>>>>>>> replication id of a >>>>>>>>>>>>>> server that no longer exists to try the cleanallruv step >>>>>>>>>>>>>> again. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is quite dangerous though. If you run cleanruv >>>>>>>>>>>>>> against a >>>>>>>>>>>>>> replica id that >>>>>>>>>>>>>> does exist it can cause a loss of data. I believe I've >>>>>>>>>>>>>> put in >>>>>>>>>>>>>> enough scary >>>>>>>>>>>>>> warnings about this. >>>>>>>>>>>>>> >>>>>>>>>>>>>> rob >>>>>>>>>>>>>> >>>>>>>>>>>>> Good work there, this should make cleaning RUVs much >>>>>>>>>>>>> easier than >>>>>>>>>>>>> with the >>>>>>>>>>>>> previous version. >>>>>>>>>>>>> >>>>>>>>>>>>> This is what I found during review: >>>>>>>>>>>>> >>>>>>>>>>>>> 1) list_ruv and clean_ruv command help in man is quite >>>>>>>>>>>>> lost. I >>>>>>>>>>>>> think >>>>>>>>>>>>> it would >>>>>>>>>>>>> help if we for example have all info for commands >>>>>>>>>>>>> indented. This >>>>>>>>>>>>> way >>>>>>>>>>>>> user could >>>>>>>>>>>>> simply over-look the new commands in the man page. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 2) I would rename new commands to clean-ruv and list-ruv >>>>>>>>>>>>> to make >>>>>>>>>>>>> them >>>>>>>>>>>>> consistent with the rest of the commands (re-initialize, >>>>>>>>>>>>> force-sync). >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 3) It would be nice to be able to run clean_ruv command in an >>>>>>>>>>>>> unattended way >>>>>>>>>>>>> (for better testing), i.e. respect --force option as we >>>>>>>>>>>>> already >>>>>>>>>>>>> do for >>>>>>>>>>>>> ipa-replica-manage del. This fix would aid test automation >>>>>>>>>>>>> in the >>>>>>>>>>>>> future. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 4) (minor) The new question (and the del too) does not >>>>>>>>>>>>> react too >>>>>>>>>>>>> well for >>>>>>>>>>>>> CTRL+D: >>>>>>>>>>>>> >>>>>>>>>>>>> # ipa-replica-manage clean_ruv 3 --force >>>>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>>>> >>>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>>>> Continue to clean? [no]: unexpected error: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 5) Help for clean_ruv command without a required parameter >>>>>>>>>>>>> is quite >>>>>>>>>>>>> confusing >>>>>>>>>>>>> as it reports that command is wrong and not the parameter: >>>>>>>>>>>>> >>>>>>>>>>>>> # ipa-replica-manage clean_ruv >>>>>>>>>>>>> Usage: ipa-replica-manage [options] >>>>>>>>>>>>> >>>>>>>>>>>>> ipa-replica-manage: error: must provide a command >>>>>>>>>>>>> [clean_ruv | >>>>>>>>>>>>> force-sync | >>>>>>>>>>>>> disconnect | connect | del | re-initialize | list | list_ruv] >>>>>>>>>>>>> >>>>>>>>>>>>> It seems you just forgot to specify the error message in the >>>>>>>>>>>>> command >>>>>>>>>>>>> definition >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 6) When the remote replica is down, the clean_ruv command >>>>>>>>>>>>> fails >>>>>>>>>>>>> with an >>>>>>>>>>>>> unexpected error: >>>>>>>>>>>>> >>>>>>>>>>>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 >>>>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>>>> >>>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>>>> Continue to clean? [no]: y >>>>>>>>>>>>> unexpected error: {'desc': 'Operations error'} >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: >>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>> cleanAllRUV_task: failed >>>>>>>>>>>>> to connect to repl agreement connection >>>>>>>>>>>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, >>>>>>>>>>>>> >>>>>>>>>>>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> tree,cn=config), error 105 >>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>> cleanAllRUV_task: replica >>>>>>>>>>>>> (cn=meTovm-055.idm.lab. >>>>>>>>>>>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> tree, cn=config) has not been cleaned. You will need to >>>>>>>>>>>>> rerun >>>>>>>>>>>>> the >>>>>>>>>>>>> CLEANALLRUV task on this replica. >>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>> cleanAllRUV_task: Task >>>>>>>>>>>>> failed (1) >>>>>>>>>>>>> >>>>>>>>>>>>> In this case I think we should inform user that the >>>>>>>>>>>>> command failed, >>>>>>>>>>>>> possibly >>>>>>>>>>>>> because of disconnected replicas and that they could >>>>>>>>>>>>> enable the >>>>>>>>>>>>> replicas and >>>>>>>>>>>>> try again. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 7) (minor) "pass" is now redundant in replication.py: >>>>>>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>>>>>>> + # We can't make the server we're removing >>>>>>>>>>>>> read-only >>>>>>>>>>>>> but >>>>>>>>>>>>> + # this isn't a show-stopper >>>>>>>>>>>>> + root_logger.debug("No permission to switch >>>>>>>>>>>>> replica to >>>>>>>>>>>>> read-only, >>>>>>>>>>>>> continuing anyway") >>>>>>>>>>>>> + pass >>>>>>>>>>>>> >>>>>>>>>>>> I think this addresses everything. >>>>>>>>>>>> >>>>>>>>>>>> rob >>>>>>>>>>> Thanks, almost there! I just found one more issue which >>>>>>>>>>> needs to be >>>>>>>>>>> fixed >>>>>>>>>>> before we push: >>>>>>>>>>> >>>>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>>>>>>>>> Directory Manager password: >>>>>>>>>>> >>>>>>>>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, >>>>>>>>>>> forcing >>>>>>>>>>> removal >>>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': >>>>>>>>>>> {'desc': >>>>>>>>>>> "Can't >>>>>>>>>>> contact LDAP server"} >>>>>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>>>>> >>>>>>>>>>> There were issues removing a connection: %d format: a number is >>>>>>>>>>> required, not str >>>>>>>>>>> >>>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': >>>>>>>>>>> {'desc': >>>>>>>>>>> "Can't >>>>>>>>>>> contact LDAP server"} >>>>>>>>>>> >>>>>>>>>>> This is a traceback I retrieved: >>>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>> File "/sbin/ipa-replica-manage", line 425, in del_master >>>>>>>>>>> del_link(realm, r, hostname, options.dirman_passwd, >>>>>>>>>>> force=True) >>>>>>>>>>> File "/sbin/ipa-replica-manage", line 271, in del_link >>>>>>>>>>> repl1.cleanallruv(replica_id) >>>>>>>>>>> File >>>>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>>>>>>>> >>>>>>>>>>> line 1094, in cleanallruv >>>>>>>>>>> root_logger.debug("Creating CLEANALLRUV task for >>>>>>>>>>> replica id >>>>>>>>>>> %d" % >>>>>>>>>>> replicaId) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> The problem here is that you don't convert replica_id to int >>>>>>>>>>> in this >>>>>>>>>>> part: >>>>>>>>>>> + replica_id = None >>>>>>>>>>> + if repl2: >>>>>>>>>>> + replica_id = repl2._get_replica_id(repl2.conn, None) >>>>>>>>>>> + else: >>>>>>>>>>> + servers = get_ruv(realm, replica1, dirman_passwd) >>>>>>>>>>> + for (netloc, rid) in servers: >>>>>>>>>>> + if netloc.startswith(replica2): >>>>>>>>>>> + replica_id = rid >>>>>>>>>>> + break >>>>>>>>>>> >>>>>>>>>>> Martin >>>>>>>>>>> >>>>>>>>>> Updated patch using new mechanism in 389-ds-base. This should >>>>>>>>>> more >>>>>>>>>> thoroughly clean out RUV data when a replica is being >>>>>>>>>> deleted, and >>>>>>>>>> provide for a way to delete RUV data afterwards too if >>>>>>>>>> necessary. >>>>>>>>>> >>>>>>>>>> rob >>>>>>>>> Rebased patch >>>>>>>>> >>>>>>>>> rob >>>>>>>>> >>>>>>>> 0) As I wrote in a review for your patch 1041, changelog entry >>>>>>>> slipped >>>>>>>> elsewhere. >>>>>>>> >>>>>>>> 1) The following KeyboardInterrupt except class looks >>>>>>>> suspicious. I >>>>>>>> know why >>>>>>>> you have it there, but since it is generally a bad thing to do, >>>>>>>> some >>>>>>>> comment >>>>>>>> why it is needed would be useful. >>>>>>>> >>>>>>>> @@ -256,6 +263,17 @@ def del_link(realm, replica1, replica2, >>>>>>>> dirman_passwd, >>>>>>>> force=False): >>>>>>>> repl1.delete_agreement(replica2) >>>>>>>> repl1.delete_referral(replica2) >>>>>>>> >>>>>>>> + if type1 == replication.IPA_REPLICA: >>>>>>>> + if repl2: >>>>>>>> + ruv = repl2._get_replica_id(repl2.conn, None) >>>>>>>> + else: >>>>>>>> + ruv = get_ruv_by_host(realm, replica1, replica2, >>>>>>>> dirman_passwd) >>>>>>>> + >>>>>>>> + try: >>>>>>>> + repl1.cleanallruv(ruv) >>>>>>>> + except KeyboardInterrupt: >>>>>>>> + pass >>>>>>>> + >>>>>>>> >>>>>>>> Maybe you just wanted to do some cleanup and then "raise" again? >>>>>>> No, it is there because it is safe to break out of it. The task >>>>>>> will >>>>>>> continue to run. I added some verbiage. >>>>>>> >>>>>>>> 2) This is related to 1), but when some replica is down, >>>>>>>> "ipa-replica-manage >>>>>>>> del" may wait indefinitely when some remote replica is down, >>>>>>>> right? >>>>>>>> >>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>>>> Deleting a master is irreversible. >>>>>>>> To reconnect to the remote master you will need to prepare a new >>>>>>>> replica file >>>>>>>> and re-install. >>>>>>>> Continue to delete? [no]: y >>>>>>>> ipa: INFO: Setting agreement >>>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>>>>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> tree,cn=config >>>>>>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 >>>>>>>> Replica >>>>>>>> acquired >>>>>>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>>>>>> Background task created to clean replication data >>>>>>>> >>>>>>>> ... after about a minute I hit CTRL+C >>>>>>>> >>>>>>>> ^CDeleted replication agreement from >>>>>>>> 'vm-086.idm.lab.bos.redhat.com' to >>>>>>>> 'vm-055.idm.lab.bos.redhat.com' >>>>>>>> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: NS >>>>>>>> record >>>>>>>> does not >>>>>>>> contain 'vm-055.idm.lab.bos.redhat.com.' >>>>>>>> You may need to manually remove them from the tree >>>>>>>> >>>>>>>> I think it would be better to inform user that some remote >>>>>>>> replica is >>>>>>>> down or >>>>>>>> at least that we are waiting for the task to complete. >>>>>>>> Something like >>>>>>>> that: >>>>>>>> >>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>>>> ... >>>>>>>> Background task created to clean replication data >>>>>>>> Replication data clean up may take very long time if some >>>>>>>> replica is >>>>>>>> unreachable >>>>>>>> Hit CTRL+C to interrupt the wait >>>>>>>> ^C Clean up wait interrupted >>>>>>>> .... >>>>>>>> [continue with del] >>>>>>> Yup, did this in #1. >>>>>>> >>>>>>>> 3) (minor) When there is a cleanruv task running and you run >>>>>>>> "ipa-replica-manage del", there is a unexpected error message with >>>>>>>> duplicate >>>>>>>> task object in LDAP: >>>>>>>> >>>>>>>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com --force >>>>>>>> Unable to connect to replica vm-072.idm.lab.bos.redhat.com, >>>>>>>> forcing >>>>>>>> removal >>>>>>>> FAIL >>>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': >>>>>>>> {'desc': "Can't >>>>>>>> contact LDAP server"} >>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>> >>>>>>>> There were issues removing a connection: This entry already exists >>>>>>>> <<<<<<<<< >>>>>>>> >>>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': >>>>>>>> {'desc': "Can't >>>>>>>> contact LDAP server"} >>>>>>>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS >>>>>>>> record >>>>>>>> does not >>>>>>>> contain 'vm-072.idm.lab.bos.redhat.com.' >>>>>>>> You may need to manually remove them from the tree >>>>>>>> >>>>>>>> >>>>>>>> I think it should be enough to just catch for "entry already >>>>>>>> exists" in >>>>>>>> cleanallruv function, and in such case print a relevant error >>>>>>>> message >>>>>>>> bail out. >>>>>>>> Thus, self.conn.checkTask(dn, dowait=True) would not be called >>>>>>>> too. >>>>>>> Good catch, fixed. >>>>>>> >>>>>>>> >>>>>>>> 4) (minor): In make_readonly function, there is a redundant "pass" >>>>>>>> statement: >>>>>>>> >>>>>>>> + def make_readonly(self): >>>>>>>> + """ >>>>>>>> + Make the current replication agreement read-only. >>>>>>>> + """ >>>>>>>> + dn = DN(('cn', 'userRoot'), ('cn', 'ldbm database'), >>>>>>>> + ('cn', 'plugins'), ('cn', 'config')) >>>>>>>> + >>>>>>>> + mod = [(ldap.MOD_REPLACE, 'nsslapd-readonly', 'on')] >>>>>>>> + try: >>>>>>>> + self.conn.modify_s(dn, mod) >>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>> + # We can't make the server we're removing >>>>>>>> read-only but >>>>>>>> + # this isn't a show-stopper >>>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>>> read-only, >>>>>>>> continuing anyway") >>>>>>>> + pass<<<<<<<<<<<<<<< >>>>>>> Yeah, this is one of my common mistakes. I put in a pass >>>>>>> initially, then >>>>>>> add logging in front of it and forget to delete the pass. Its >>>>>>> gone now. >>>>>>> >>>>>>>> >>>>>>>> 5) In clean_ruv, I think allowing a --force option to bypass the >>>>>>>> user_input >>>>>>>> would be helpful (at least for test automation): >>>>>>>> >>>>>>>> + if not ipautil.user_input("Continue to clean?", False): >>>>>>>> + sys.exit("Aborted") >>>>>>> Yup, added. >>>>>>> >>>>>>> rob >>>>>> Slightly revised patch. I still had a window open with one >>>>>> unsaved change. >>>>>> >>>>>> rob >>>>>> >>>>> Apparently there were two unsaved changes, one of which was lost. >>>>> This adds in >>>>> the 'entry already exists' fix. >>>>> >>>>> rob >>>>> >>>> Just one last thing (otherwise the patch is OK) - I don't think >>>> this is what we >>>> want :-) >>>> >>>> # ipa-replica-manage clean-ruv 8 >>>> Clean the Replication Update Vector for >>>> vm-055.idm.lab.bos.redhat.com:389 >>>> >>>> Cleaning the wrong replica ID will cause that server to no >>>> longer replicate so it may miss updates while the process >>>> is running. It would need to be re-initialized to maintain >>>> consistency. Be very careful. >>>> Continue to clean? [no]: y<<<<<< >>>> Aborted >>>> >>>> >>>> Nor this exception, (your are checking for wrong exception): >>>> >>>> # ipa-replica-manage clean-ruv 8 >>>> Clean the Replication Update Vector for >>>> vm-055.idm.lab.bos.redhat.com:389 >>>> >>>> Cleaning the wrong replica ID will cause that server to no >>>> longer replicate so it may miss updates while the process >>>> is running. It would need to be re-initialized to maintain >>>> consistency. Be very careful. >>>> Continue to clean? [no]: >>>> unexpected error: This entry already exists >>>> >>>> This is the exception: >>>> >>>> Traceback (most recent call last): >>>> File "/sbin/ipa-replica-manage", line 651, in >>>> main() >>>> File "/sbin/ipa-replica-manage", line 648, in main >>>> clean_ruv(realm, args[1], options) >>>> File "/sbin/ipa-replica-manage", line 373, in clean_ruv >>>> thisrepl.cleanallruv(ruv) >>>> File >>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>> line 1136, in cleanallruv >>>> self.conn.addEntry(e) >>>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", >>>> line 503, in >>>> addEntry >>>> self.__handle_errors(e, arg_desc=arg_desc) >>>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", >>>> line 321, in >>>> __handle_errors >>>> raise errors.DuplicateEntry() >>>> ipalib.errors.DuplicateEntry: This entry already exists >>>> >>>> Martin >>>> >>> >>> On another matter, I just noticed that CLEANRUV is not proceeding if >>> I have a >>> winsync replica defined (and it is even up): >>> >>> # ipa-replica-manage list >>> dc.ad.test: winsync<<<<<<< >>> vm-072.idm.lab.bos.redhat.com: master >>> vm-086.idm.lab.bos.redhat.com: master >>> >>> [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV >>> Task: Waiting >>> for all the replicas to receive all the deleted replica updates... >>> [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV >>> Task: Failed >>> to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), will >>> retry later. >>> [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV >>> Task: Not all >>> replicas caught up, retrying in 10 seconds >>> [06/Sep/2012:11:59:20 -0400] NSMMReplicationPlugin - CleanAllRUV >>> Task: Failed >>> to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), will >>> retry later. >>> [06/Sep/2012:11:59:20 -0400] NSMMReplicationPlugin - CleanAllRUV >>> Task: Not all >>> replicas caught up, retrying in 20 seconds >>> [06/Sep/2012:11:59:40 -0400] NSMMReplicationPlugin - CleanAllRUV >>> Task: Failed >>> to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), will >>> retry later. >>> [06/Sep/2012:11:59:40 -0400] NSMMReplicationPlugin - CleanAllRUV >>> Task: Not all >>> replicas caught up, retrying in 40 seconds >>> >>> I don't think this is OK. Adding Rich to CC to follow on this one. >>> >>> Martin >>> >> And now the actual CC. >> Martin >> > Yeah - looks like CLEANALLRUV needs to ignore windows sync agreements Yeah, sorry not that familiar with winsync(didn't know it used the same repl agmts). I will have to do another fix... > > _______________________________________________ > Freeipa-devel mailing list > Freeipa-devel at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-devel -- Mark Reynolds Senior Software Engineer Red Hat, Inc mreynolds at redhat.com From mareynol at redhat.com Thu Sep 6 16:42:40 2012 From: mareynol at redhat.com (Mark Reynolds) Date: Thu, 06 Sep 2012 12:42:40 -0400 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <5048CEE5.80704@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> <50479491.1060200@redhat.com> <50488DD0.1080507@redhat.com> <5048BEF8.5000401@redhat.com> <5048C46A.8020207@redhat.com> <5048C754.2030708@redhat.com> <5048C9BE.1050005@redhat.com> <5048CA9F.7060607@redhat.com> <5048CAD0.2070508@redhat.com> <5048CB9E.4090301@redhat.com> <5048CEE5.80704@redhat.com> Message-ID: <5048D280.2050607@redhat.com> On 09/06/2012 12:27 PM, Martin Kosek wrote: > On 09/06/2012 06:13 PM, Rich Megginson wrote: >> On 09/06/2012 10:09 AM, Martin Kosek wrote: >>> On 09/06/2012 06:09 PM, Martin Kosek wrote: >>>> On 09/06/2012 06:05 PM, Martin Kosek wrote: >>>>> On 09/06/2012 05:55 PM, Rob Crittenden wrote: >>>>>> Rob Crittenden wrote: >>>>>>> Rob Crittenden wrote: >>>>>>>> Martin Kosek wrote: >>>>>>>>> On 09/05/2012 08:06 PM, Rob Crittenden wrote: >>>>>>>>>> Rob Crittenden wrote: >>>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>>> On 07/05/2012 08:39 PM, Rob Crittenden wrote: >>>>>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>>>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: >>>>>>>>>>>>>>> Deleting a replica can leave a replication vector (RUV) on the >>>>>>>>>>>>>>> other servers. >>>>>>>>>>>>>>> This can confuse things if the replica is re-added, and it also >>>>>>>>>>>>>>> causes the >>>>>>>>>>>>>>> server to calculate changes against a server that may no longer >>>>>>>>>>>>>>> exist. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 389-ds-base provides a new task that self-propogates itself to all >>>>>>>>>>>>>>> available >>>>>>>>>>>>>>> replicas to clean this RUV data. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This patch will create this task at deletion time to hopefully >>>>>>>>>>>>>>> clean things up. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It isn't perfect. If any replica is down or unavailable at the >>>>>>>>>>>>>>> time >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> cleanruv task fires, and then comes back up, the old RUV data >>>>>>>>>>>>>>> may be >>>>>>>>>>>>>>> re-propogated around. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> To make things easier in this case I've added two new commands to >>>>>>>>>>>>>>> ipa-replica-manage. The first lists the replication ids of all the >>>>>>>>>>>>>>> servers we >>>>>>>>>>>>>>> have a RUV for. Using this you can call clean_ruv with the >>>>>>>>>>>>>>> replication id of a >>>>>>>>>>>>>>> server that no longer exists to try the cleanallruv step again. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is quite dangerous though. If you run cleanruv against a >>>>>>>>>>>>>>> replica id that >>>>>>>>>>>>>>> does exist it can cause a loss of data. I believe I've put in >>>>>>>>>>>>>>> enough scary >>>>>>>>>>>>>>> warnings about this. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> rob >>>>>>>>>>>>>>> >>>>>>>>>>>>>> Good work there, this should make cleaning RUVs much easier than >>>>>>>>>>>>>> with the >>>>>>>>>>>>>> previous version. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is what I found during review: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1) list_ruv and clean_ruv command help in man is quite lost. I >>>>>>>>>>>>>> think >>>>>>>>>>>>>> it would >>>>>>>>>>>>>> help if we for example have all info for commands indented. This >>>>>>>>>>>>>> way >>>>>>>>>>>>>> user could >>>>>>>>>>>>>> simply over-look the new commands in the man page. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2) I would rename new commands to clean-ruv and list-ruv to make >>>>>>>>>>>>>> them >>>>>>>>>>>>>> consistent with the rest of the commands (re-initialize, >>>>>>>>>>>>>> force-sync). >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 3) It would be nice to be able to run clean_ruv command in an >>>>>>>>>>>>>> unattended way >>>>>>>>>>>>>> (for better testing), i.e. respect --force option as we already >>>>>>>>>>>>>> do for >>>>>>>>>>>>>> ipa-replica-manage del. This fix would aid test automation in the >>>>>>>>>>>>>> future. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 4) (minor) The new question (and the del too) does not react too >>>>>>>>>>>>>> well for >>>>>>>>>>>>>> CTRL+D: >>>>>>>>>>>>>> >>>>>>>>>>>>>> # ipa-replica-manage clean_ruv 3 --force >>>>>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>>>>> Continue to clean? [no]: unexpected error: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 5) Help for clean_ruv command without a required parameter is quite >>>>>>>>>>>>>> confusing >>>>>>>>>>>>>> as it reports that command is wrong and not the parameter: >>>>>>>>>>>>>> >>>>>>>>>>>>>> # ipa-replica-manage clean_ruv >>>>>>>>>>>>>> Usage: ipa-replica-manage [options] >>>>>>>>>>>>>> >>>>>>>>>>>>>> ipa-replica-manage: error: must provide a command [clean_ruv | >>>>>>>>>>>>>> force-sync | >>>>>>>>>>>>>> disconnect | connect | del | re-initialize | list | list_ruv] >>>>>>>>>>>>>> >>>>>>>>>>>>>> It seems you just forgot to specify the error message in the >>>>>>>>>>>>>> command >>>>>>>>>>>>>> definition >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 6) When the remote replica is down, the clean_ruv command fails >>>>>>>>>>>>>> with an >>>>>>>>>>>>>> unexpected error: >>>>>>>>>>>>>> >>>>>>>>>>>>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 >>>>>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>>>>> Continue to clean? [no]: y >>>>>>>>>>>>>> unexpected error: {'desc': 'Operations error'} >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: >>>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>>> cleanAllRUV_task: failed >>>>>>>>>>>>>> to connect to repl agreement connection >>>>>>>>>>>>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, >>>>>>>>>>>>>> >>>>>>>>>>>>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> tree,cn=config), error 105 >>>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>>> cleanAllRUV_task: replica >>>>>>>>>>>>>> (cn=meTovm-055.idm.lab. >>>>>>>>>>>>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> tree, cn=config) has not been cleaned. You will need to rerun >>>>>>>>>>>>>> the >>>>>>>>>>>>>> CLEANALLRUV task on this replica. >>>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>>> cleanAllRUV_task: Task >>>>>>>>>>>>>> failed (1) >>>>>>>>>>>>>> >>>>>>>>>>>>>> In this case I think we should inform user that the command failed, >>>>>>>>>>>>>> possibly >>>>>>>>>>>>>> because of disconnected replicas and that they could enable the >>>>>>>>>>>>>> replicas and >>>>>>>>>>>>>> try again. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 7) (minor) "pass" is now redundant in replication.py: >>>>>>>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>>>>>>>> + # We can't make the server we're removing read-only >>>>>>>>>>>>>> but >>>>>>>>>>>>>> + # this isn't a show-stopper >>>>>>>>>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>>>>>>>>> read-only, >>>>>>>>>>>>>> continuing anyway") >>>>>>>>>>>>>> + pass >>>>>>>>>>>>>> >>>>>>>>>>>>> I think this addresses everything. >>>>>>>>>>>>> >>>>>>>>>>>>> rob >>>>>>>>>>>> Thanks, almost there! I just found one more issue which needs to be >>>>>>>>>>>> fixed >>>>>>>>>>>> before we push: >>>>>>>>>>>> >>>>>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>>>>>>>>>> Directory Manager password: >>>>>>>>>>>> >>>>>>>>>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing >>>>>>>>>>>> removal >>>>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>>>>>> "Can't >>>>>>>>>>>> contact LDAP server"} >>>>>>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>>>>>> >>>>>>>>>>>> There were issues removing a connection: %d format: a number is >>>>>>>>>>>> required, not str >>>>>>>>>>>> >>>>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>>>>>> "Can't >>>>>>>>>>>> contact LDAP server"} >>>>>>>>>>>> >>>>>>>>>>>> This is a traceback I retrieved: >>>>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>>> File "/sbin/ipa-replica-manage", line 425, in del_master >>>>>>>>>>>> del_link(realm, r, hostname, options.dirman_passwd, force=True) >>>>>>>>>>>> File "/sbin/ipa-replica-manage", line 271, in del_link >>>>>>>>>>>> repl1.cleanallruv(replica_id) >>>>>>>>>>>> File >>>>>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>>>>>>>>> line 1094, in cleanallruv >>>>>>>>>>>> root_logger.debug("Creating CLEANALLRUV task for replica id >>>>>>>>>>>> %d" % >>>>>>>>>>>> replicaId) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> The problem here is that you don't convert replica_id to int in this >>>>>>>>>>>> part: >>>>>>>>>>>> + replica_id = None >>>>>>>>>>>> + if repl2: >>>>>>>>>>>> + replica_id = repl2._get_replica_id(repl2.conn, None) >>>>>>>>>>>> + else: >>>>>>>>>>>> + servers = get_ruv(realm, replica1, dirman_passwd) >>>>>>>>>>>> + for (netloc, rid) in servers: >>>>>>>>>>>> + if netloc.startswith(replica2): >>>>>>>>>>>> + replica_id = rid >>>>>>>>>>>> + break >>>>>>>>>>>> >>>>>>>>>>>> Martin >>>>>>>>>>>> >>>>>>>>>>> Updated patch using new mechanism in 389-ds-base. This should more >>>>>>>>>>> thoroughly clean out RUV data when a replica is being deleted, and >>>>>>>>>>> provide for a way to delete RUV data afterwards too if necessary. >>>>>>>>>>> >>>>>>>>>>> rob >>>>>>>>>> Rebased patch >>>>>>>>>> >>>>>>>>>> rob >>>>>>>>>> >>>>>>>>> 0) As I wrote in a review for your patch 1041, changelog entry slipped >>>>>>>>> elsewhere. >>>>>>>>> >>>>>>>>> 1) The following KeyboardInterrupt except class looks suspicious. I >>>>>>>>> know why >>>>>>>>> you have it there, but since it is generally a bad thing to do, some >>>>>>>>> comment >>>>>>>>> why it is needed would be useful. >>>>>>>>> >>>>>>>>> @@ -256,6 +263,17 @@ def del_link(realm, replica1, replica2, >>>>>>>>> dirman_passwd, >>>>>>>>> force=False): >>>>>>>>> repl1.delete_agreement(replica2) >>>>>>>>> repl1.delete_referral(replica2) >>>>>>>>> >>>>>>>>> + if type1 == replication.IPA_REPLICA: >>>>>>>>> + if repl2: >>>>>>>>> + ruv = repl2._get_replica_id(repl2.conn, None) >>>>>>>>> + else: >>>>>>>>> + ruv = get_ruv_by_host(realm, replica1, replica2, >>>>>>>>> dirman_passwd) >>>>>>>>> + >>>>>>>>> + try: >>>>>>>>> + repl1.cleanallruv(ruv) >>>>>>>>> + except KeyboardInterrupt: >>>>>>>>> + pass >>>>>>>>> + >>>>>>>>> >>>>>>>>> Maybe you just wanted to do some cleanup and then "raise" again? >>>>>>>> No, it is there because it is safe to break out of it. The task will >>>>>>>> continue to run. I added some verbiage. >>>>>>>> >>>>>>>>> 2) This is related to 1), but when some replica is down, >>>>>>>>> "ipa-replica-manage >>>>>>>>> del" may wait indefinitely when some remote replica is down, right? >>>>>>>>> >>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>>>>> Deleting a master is irreversible. >>>>>>>>> To reconnect to the remote master you will need to prepare a new >>>>>>>>> replica file >>>>>>>>> and re-install. >>>>>>>>> Continue to delete? [no]: y >>>>>>>>> ipa: INFO: Setting agreement >>>>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>>>>>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> tree,cn=config >>>>>>>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica >>>>>>>>> acquired >>>>>>>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>>>>>>> Background task created to clean replication data >>>>>>>>> >>>>>>>>> ... after about a minute I hit CTRL+C >>>>>>>>> >>>>>>>>> ^CDeleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to >>>>>>>>> 'vm-055.idm.lab.bos.redhat.com' >>>>>>>>> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: NS record >>>>>>>>> does not >>>>>>>>> contain 'vm-055.idm.lab.bos.redhat.com.' >>>>>>>>> You may need to manually remove them from the tree >>>>>>>>> >>>>>>>>> I think it would be better to inform user that some remote replica is >>>>>>>>> down or >>>>>>>>> at least that we are waiting for the task to complete. Something like >>>>>>>>> that: >>>>>>>>> >>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>>>>> ... >>>>>>>>> Background task created to clean replication data >>>>>>>>> Replication data clean up may take very long time if some replica is >>>>>>>>> unreachable >>>>>>>>> Hit CTRL+C to interrupt the wait >>>>>>>>> ^C Clean up wait interrupted >>>>>>>>> .... >>>>>>>>> [continue with del] >>>>>>>> Yup, did this in #1. >>>>>>>> >>>>>>>>> 3) (minor) When there is a cleanruv task running and you run >>>>>>>>> "ipa-replica-manage del", there is a unexpected error message with >>>>>>>>> duplicate >>>>>>>>> task object in LDAP: >>>>>>>>> >>>>>>>>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com --force >>>>>>>>> Unable to connect to replica vm-072.idm.lab.bos.redhat.com, forcing >>>>>>>>> removal >>>>>>>>> FAIL >>>>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>>>>>> contact LDAP server"} >>>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>>> >>>>>>>>> There were issues removing a connection: This entry already exists >>>>>>>>> <<<<<<<<< >>>>>>>>> >>>>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>>>>>> contact LDAP server"} >>>>>>>>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record >>>>>>>>> does not >>>>>>>>> contain 'vm-072.idm.lab.bos.redhat.com.' >>>>>>>>> You may need to manually remove them from the tree >>>>>>>>> >>>>>>>>> >>>>>>>>> I think it should be enough to just catch for "entry already exists" in >>>>>>>>> cleanallruv function, and in such case print a relevant error message >>>>>>>>> bail out. >>>>>>>>> Thus, self.conn.checkTask(dn, dowait=True) would not be called too. >>>>>>>> Good catch, fixed. >>>>>>>> >>>>>>>>> 4) (minor): In make_readonly function, there is a redundant "pass" >>>>>>>>> statement: >>>>>>>>> >>>>>>>>> + def make_readonly(self): >>>>>>>>> + """ >>>>>>>>> + Make the current replication agreement read-only. >>>>>>>>> + """ >>>>>>>>> + dn = DN(('cn', 'userRoot'), ('cn', 'ldbm database'), >>>>>>>>> + ('cn', 'plugins'), ('cn', 'config')) >>>>>>>>> + >>>>>>>>> + mod = [(ldap.MOD_REPLACE, 'nsslapd-readonly', 'on')] >>>>>>>>> + try: >>>>>>>>> + self.conn.modify_s(dn, mod) >>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>>> + # We can't make the server we're removing read-only but >>>>>>>>> + # this isn't a show-stopper >>>>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>>>> read-only, >>>>>>>>> continuing anyway") >>>>>>>>> + pass<<<<<<<<<<<<<<< >>>>>>>> Yeah, this is one of my common mistakes. I put in a pass initially, then >>>>>>>> add logging in front of it and forget to delete the pass. Its gone now. >>>>>>>> >>>>>>>>> 5) In clean_ruv, I think allowing a --force option to bypass the >>>>>>>>> user_input >>>>>>>>> would be helpful (at least for test automation): >>>>>>>>> >>>>>>>>> + if not ipautil.user_input("Continue to clean?", False): >>>>>>>>> + sys.exit("Aborted") >>>>>>>> Yup, added. >>>>>>>> >>>>>>>> rob >>>>>>> Slightly revised patch. I still had a window open with one unsaved change. >>>>>>> >>>>>>> rob >>>>>>> >>>>>> Apparently there were two unsaved changes, one of which was lost. This >>>>>> adds in >>>>>> the 'entry already exists' fix. >>>>>> >>>>>> rob >>>>>> >>>>> Just one last thing (otherwise the patch is OK) - I don't think this is >>>>> what we >>>>> want :-) >>>>> >>>>> # ipa-replica-manage clean-ruv 8 >>>>> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 >>>>> >>>>> Cleaning the wrong replica ID will cause that server to no >>>>> longer replicate so it may miss updates while the process >>>>> is running. It would need to be re-initialized to maintain >>>>> consistency. Be very careful. >>>>> Continue to clean? [no]: y<<<<<< >>>>> Aborted >>>>> >>>>> >>>>> Nor this exception, (your are checking for wrong exception): >>>>> >>>>> # ipa-replica-manage clean-ruv 8 >>>>> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 >>>>> >>>>> Cleaning the wrong replica ID will cause that server to no >>>>> longer replicate so it may miss updates while the process >>>>> is running. It would need to be re-initialized to maintain >>>>> consistency. Be very careful. >>>>> Continue to clean? [no]: >>>>> unexpected error: This entry already exists >>>>> >>>>> This is the exception: >>>>> >>>>> Traceback (most recent call last): >>>>> File "/sbin/ipa-replica-manage", line 651, in >>>>> main() >>>>> File "/sbin/ipa-replica-manage", line 648, in main >>>>> clean_ruv(realm, args[1], options) >>>>> File "/sbin/ipa-replica-manage", line 373, in clean_ruv >>>>> thisrepl.cleanallruv(ruv) >>>>> File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>> line 1136, in cleanallruv >>>>> self.conn.addEntry(e) >>>>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 503, in >>>>> addEntry >>>>> self.__handle_errors(e, arg_desc=arg_desc) >>>>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 321, in >>>>> __handle_errors >>>>> raise errors.DuplicateEntry() >>>>> ipalib.errors.DuplicateEntry: This entry already exists >>>>> >>>>> Martin >>>>> >>>> On another matter, I just noticed that CLEANRUV is not proceeding if I have a >>>> winsync replica defined (and it is even up): >>>> >>>> # ipa-replica-manage list >>>> dc.ad.test: winsync<<<<<<< >>>> vm-072.idm.lab.bos.redhat.com: master >>>> vm-086.idm.lab.bos.redhat.com: master >>>> >>>> [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Waiting >>>> for all the replicas to receive all the deleted replica updates... >>>> [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Failed >>>> to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), will retry >>>> later. >>>> [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all >>>> replicas caught up, retrying in 10 seconds >>>> [06/Sep/2012:11:59:20 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Failed >>>> to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), will retry >>>> later. >>>> [06/Sep/2012:11:59:20 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all >>>> replicas caught up, retrying in 20 seconds >>>> [06/Sep/2012:11:59:40 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Failed >>>> to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), will retry >>>> later. >>>> [06/Sep/2012:11:59:40 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all >>>> replicas caught up, retrying in 40 seconds >>>> >>>> I don't think this is OK. Adding Rich to CC to follow on this one. >>>> >>>> Martin >>>> >>> And now the actual CC. >>> Martin >>> >> Yeah - looks like CLEANALLRUV needs to ignore windows sync agreements > Have you created a 389 ticket for that or would you like me to do that? If you could, that would be great. > > While talking about this issue, does DS also cover a situation when one of the > replicas does not support CLEANRUV task? If it doesn't support the new CLEANALLRUV task, it will manually add the old CLEANRUV task to that replica. > Would not CLEANRUV stumble similarly > to winsync agreement? It might, but I will have to check for winsync agreements and ignore them. So it should not be an issue moving forward. > > Martin > > _______________________________________________ > Freeipa-devel mailing list > Freeipa-devel at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-devel -- Mark Reynolds Senior Software Engineer Red Hat, Inc mreynolds at redhat.com From rmeggins at redhat.com Thu Sep 6 16:44:41 2012 From: rmeggins at redhat.com (Rich Megginson) Date: Thu, 06 Sep 2012 10:44:41 -0600 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <5048D203.30807@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> <50479491.1060200@redhat.com> <50488DD0.1080507@redhat.com> <5048BEF8.5000401@redhat.com> <5048C46A.8020207@redhat.com> <5048C754.2030708@redhat.com> <5048C9BE.1050005@redhat.com> <5048CA9F.7060607@redhat.com> <5048CAD0.2070508@redhat.com> <5048CB9E.4090301@redhat.com> <5048D203.30807@redhat.com> Message-ID: <5048D2F9.9070809@redhat.com> On 09/06/2012 10:40 AM, Mark Reynolds wrote: > > > On 09/06/2012 12:13 PM, Rich Megginson wrote: >> On 09/06/2012 10:09 AM, Martin Kosek wrote: >>> On 09/06/2012 06:09 PM, Martin Kosek wrote: >>>> On 09/06/2012 06:05 PM, Martin Kosek wrote: >>>>> On 09/06/2012 05:55 PM, Rob Crittenden wrote: >>>>>> Rob Crittenden wrote: >>>>>>> Rob Crittenden wrote: >>>>>>>> Martin Kosek wrote: >>>>>>>>> On 09/05/2012 08:06 PM, Rob Crittenden wrote: >>>>>>>>>> Rob Crittenden wrote: >>>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>>> On 07/05/2012 08:39 PM, Rob Crittenden wrote: >>>>>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>>>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: >>>>>>>>>>>>>>> Deleting a replica can leave a replication vector (RUV) >>>>>>>>>>>>>>> on the >>>>>>>>>>>>>>> other servers. >>>>>>>>>>>>>>> This can confuse things if the replica is re-added, and >>>>>>>>>>>>>>> it also >>>>>>>>>>>>>>> causes the >>>>>>>>>>>>>>> server to calculate changes against a server that may no >>>>>>>>>>>>>>> longer >>>>>>>>>>>>>>> exist. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 389-ds-base provides a new task that self-propogates >>>>>>>>>>>>>>> itself to all >>>>>>>>>>>>>>> available >>>>>>>>>>>>>>> replicas to clean this RUV data. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This patch will create this task at deletion time to >>>>>>>>>>>>>>> hopefully >>>>>>>>>>>>>>> clean things up. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It isn't perfect. If any replica is down or unavailable >>>>>>>>>>>>>>> at the >>>>>>>>>>>>>>> time >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> cleanruv task fires, and then comes back up, the old RUV >>>>>>>>>>>>>>> data >>>>>>>>>>>>>>> may be >>>>>>>>>>>>>>> re-propogated around. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> To make things easier in this case I've added two new >>>>>>>>>>>>>>> commands to >>>>>>>>>>>>>>> ipa-replica-manage. The first lists the replication ids >>>>>>>>>>>>>>> of all the >>>>>>>>>>>>>>> servers we >>>>>>>>>>>>>>> have a RUV for. Using this you can call clean_ruv with the >>>>>>>>>>>>>>> replication id of a >>>>>>>>>>>>>>> server that no longer exists to try the cleanallruv step >>>>>>>>>>>>>>> again. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is quite dangerous though. If you run cleanruv >>>>>>>>>>>>>>> against a >>>>>>>>>>>>>>> replica id that >>>>>>>>>>>>>>> does exist it can cause a loss of data. I believe I've >>>>>>>>>>>>>>> put in >>>>>>>>>>>>>>> enough scary >>>>>>>>>>>>>>> warnings about this. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> rob >>>>>>>>>>>>>>> >>>>>>>>>>>>>> Good work there, this should make cleaning RUVs much >>>>>>>>>>>>>> easier than >>>>>>>>>>>>>> with the >>>>>>>>>>>>>> previous version. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is what I found during review: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1) list_ruv and clean_ruv command help in man is quite >>>>>>>>>>>>>> lost. I >>>>>>>>>>>>>> think >>>>>>>>>>>>>> it would >>>>>>>>>>>>>> help if we for example have all info for commands >>>>>>>>>>>>>> indented. This >>>>>>>>>>>>>> way >>>>>>>>>>>>>> user could >>>>>>>>>>>>>> simply over-look the new commands in the man page. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2) I would rename new commands to clean-ruv and list-ruv >>>>>>>>>>>>>> to make >>>>>>>>>>>>>> them >>>>>>>>>>>>>> consistent with the rest of the commands (re-initialize, >>>>>>>>>>>>>> force-sync). >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 3) It would be nice to be able to run clean_ruv command >>>>>>>>>>>>>> in an >>>>>>>>>>>>>> unattended way >>>>>>>>>>>>>> (for better testing), i.e. respect --force option as we >>>>>>>>>>>>>> already >>>>>>>>>>>>>> do for >>>>>>>>>>>>>> ipa-replica-manage del. This fix would aid test >>>>>>>>>>>>>> automation in the >>>>>>>>>>>>>> future. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 4) (minor) The new question (and the del too) does not >>>>>>>>>>>>>> react too >>>>>>>>>>>>>> well for >>>>>>>>>>>>>> CTRL+D: >>>>>>>>>>>>>> >>>>>>>>>>>>>> # ipa-replica-manage clean_ruv 3 --force >>>>>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>>>>> Continue to clean? [no]: unexpected error: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 5) Help for clean_ruv command without a required >>>>>>>>>>>>>> parameter is quite >>>>>>>>>>>>>> confusing >>>>>>>>>>>>>> as it reports that command is wrong and not the parameter: >>>>>>>>>>>>>> >>>>>>>>>>>>>> # ipa-replica-manage clean_ruv >>>>>>>>>>>>>> Usage: ipa-replica-manage [options] >>>>>>>>>>>>>> >>>>>>>>>>>>>> ipa-replica-manage: error: must provide a command >>>>>>>>>>>>>> [clean_ruv | >>>>>>>>>>>>>> force-sync | >>>>>>>>>>>>>> disconnect | connect | del | re-initialize | list | >>>>>>>>>>>>>> list_ruv] >>>>>>>>>>>>>> >>>>>>>>>>>>>> It seems you just forgot to specify the error message in the >>>>>>>>>>>>>> command >>>>>>>>>>>>>> definition >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 6) When the remote replica is down, the clean_ruv command >>>>>>>>>>>>>> fails >>>>>>>>>>>>>> with an >>>>>>>>>>>>>> unexpected error: >>>>>>>>>>>>>> >>>>>>>>>>>>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 >>>>>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>>>>> Continue to clean? [no]: y >>>>>>>>>>>>>> unexpected error: {'desc': 'Operations error'} >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: >>>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>>> cleanAllRUV_task: failed >>>>>>>>>>>>>> to connect to repl agreement connection >>>>>>>>>>>>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, >>>>>>>>>>>>>> >>>>>>>>>>>>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> tree,cn=config), error 105 >>>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>>> cleanAllRUV_task: replica >>>>>>>>>>>>>> (cn=meTovm-055.idm.lab. >>>>>>>>>>>>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> tree, cn=config) has not been cleaned. You will need >>>>>>>>>>>>>> to rerun >>>>>>>>>>>>>> the >>>>>>>>>>>>>> CLEANALLRUV task on this replica. >>>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>>> cleanAllRUV_task: Task >>>>>>>>>>>>>> failed (1) >>>>>>>>>>>>>> >>>>>>>>>>>>>> In this case I think we should inform user that the >>>>>>>>>>>>>> command failed, >>>>>>>>>>>>>> possibly >>>>>>>>>>>>>> because of disconnected replicas and that they could >>>>>>>>>>>>>> enable the >>>>>>>>>>>>>> replicas and >>>>>>>>>>>>>> try again. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 7) (minor) "pass" is now redundant in replication.py: >>>>>>>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>>>>>>>> + # We can't make the server we're removing >>>>>>>>>>>>>> read-only >>>>>>>>>>>>>> but >>>>>>>>>>>>>> + # this isn't a show-stopper >>>>>>>>>>>>>> + root_logger.debug("No permission to switch >>>>>>>>>>>>>> replica to >>>>>>>>>>>>>> read-only, >>>>>>>>>>>>>> continuing anyway") >>>>>>>>>>>>>> + pass >>>>>>>>>>>>>> >>>>>>>>>>>>> I think this addresses everything. >>>>>>>>>>>>> >>>>>>>>>>>>> rob >>>>>>>>>>>> Thanks, almost there! I just found one more issue which >>>>>>>>>>>> needs to be >>>>>>>>>>>> fixed >>>>>>>>>>>> before we push: >>>>>>>>>>>> >>>>>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>>>>>>>>>> Directory Manager password: >>>>>>>>>>>> >>>>>>>>>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, >>>>>>>>>>>> forcing >>>>>>>>>>>> removal >>>>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': >>>>>>>>>>>> {'desc': >>>>>>>>>>>> "Can't >>>>>>>>>>>> contact LDAP server"} >>>>>>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>>>>>> >>>>>>>>>>>> There were issues removing a connection: %d format: a >>>>>>>>>>>> number is >>>>>>>>>>>> required, not str >>>>>>>>>>>> >>>>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': >>>>>>>>>>>> {'desc': >>>>>>>>>>>> "Can't >>>>>>>>>>>> contact LDAP server"} >>>>>>>>>>>> >>>>>>>>>>>> This is a traceback I retrieved: >>>>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>>> File "/sbin/ipa-replica-manage", line 425, in del_master >>>>>>>>>>>> del_link(realm, r, hostname, options.dirman_passwd, >>>>>>>>>>>> force=True) >>>>>>>>>>>> File "/sbin/ipa-replica-manage", line 271, in del_link >>>>>>>>>>>> repl1.cleanallruv(replica_id) >>>>>>>>>>>> File >>>>>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>>>>>>>>> >>>>>>>>>>>> line 1094, in cleanallruv >>>>>>>>>>>> root_logger.debug("Creating CLEANALLRUV task for >>>>>>>>>>>> replica id >>>>>>>>>>>> %d" % >>>>>>>>>>>> replicaId) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> The problem here is that you don't convert replica_id to >>>>>>>>>>>> int in this >>>>>>>>>>>> part: >>>>>>>>>>>> + replica_id = None >>>>>>>>>>>> + if repl2: >>>>>>>>>>>> + replica_id = repl2._get_replica_id(repl2.conn, None) >>>>>>>>>>>> + else: >>>>>>>>>>>> + servers = get_ruv(realm, replica1, dirman_passwd) >>>>>>>>>>>> + for (netloc, rid) in servers: >>>>>>>>>>>> + if netloc.startswith(replica2): >>>>>>>>>>>> + replica_id = rid >>>>>>>>>>>> + break >>>>>>>>>>>> >>>>>>>>>>>> Martin >>>>>>>>>>>> >>>>>>>>>>> Updated patch using new mechanism in 389-ds-base. This >>>>>>>>>>> should more >>>>>>>>>>> thoroughly clean out RUV data when a replica is being >>>>>>>>>>> deleted, and >>>>>>>>>>> provide for a way to delete RUV data afterwards too if >>>>>>>>>>> necessary. >>>>>>>>>>> >>>>>>>>>>> rob >>>>>>>>>> Rebased patch >>>>>>>>>> >>>>>>>>>> rob >>>>>>>>>> >>>>>>>>> 0) As I wrote in a review for your patch 1041, changelog entry >>>>>>>>> slipped >>>>>>>>> elsewhere. >>>>>>>>> >>>>>>>>> 1) The following KeyboardInterrupt except class looks >>>>>>>>> suspicious. I >>>>>>>>> know why >>>>>>>>> you have it there, but since it is generally a bad thing to >>>>>>>>> do, some >>>>>>>>> comment >>>>>>>>> why it is needed would be useful. >>>>>>>>> >>>>>>>>> @@ -256,6 +263,17 @@ def del_link(realm, replica1, replica2, >>>>>>>>> dirman_passwd, >>>>>>>>> force=False): >>>>>>>>> repl1.delete_agreement(replica2) >>>>>>>>> repl1.delete_referral(replica2) >>>>>>>>> >>>>>>>>> + if type1 == replication.IPA_REPLICA: >>>>>>>>> + if repl2: >>>>>>>>> + ruv = repl2._get_replica_id(repl2.conn, None) >>>>>>>>> + else: >>>>>>>>> + ruv = get_ruv_by_host(realm, replica1, replica2, >>>>>>>>> dirman_passwd) >>>>>>>>> + >>>>>>>>> + try: >>>>>>>>> + repl1.cleanallruv(ruv) >>>>>>>>> + except KeyboardInterrupt: >>>>>>>>> + pass >>>>>>>>> + >>>>>>>>> >>>>>>>>> Maybe you just wanted to do some cleanup and then "raise" again? >>>>>>>> No, it is there because it is safe to break out of it. The task >>>>>>>> will >>>>>>>> continue to run. I added some verbiage. >>>>>>>> >>>>>>>>> 2) This is related to 1), but when some replica is down, >>>>>>>>> "ipa-replica-manage >>>>>>>>> del" may wait indefinitely when some remote replica is down, >>>>>>>>> right? >>>>>>>>> >>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>>>>> Deleting a master is irreversible. >>>>>>>>> To reconnect to the remote master you will need to prepare a new >>>>>>>>> replica file >>>>>>>>> and re-install. >>>>>>>>> Continue to delete? [no]: y >>>>>>>>> ipa: INFO: Setting agreement >>>>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>>>>>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> tree,cn=config >>>>>>>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 >>>>>>>>> Replica >>>>>>>>> acquired >>>>>>>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>>>>>>> Background task created to clean replication data >>>>>>>>> >>>>>>>>> ... after about a minute I hit CTRL+C >>>>>>>>> >>>>>>>>> ^CDeleted replication agreement from >>>>>>>>> 'vm-086.idm.lab.bos.redhat.com' to >>>>>>>>> 'vm-055.idm.lab.bos.redhat.com' >>>>>>>>> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: >>>>>>>>> NS record >>>>>>>>> does not >>>>>>>>> contain 'vm-055.idm.lab.bos.redhat.com.' >>>>>>>>> You may need to manually remove them from the tree >>>>>>>>> >>>>>>>>> I think it would be better to inform user that some remote >>>>>>>>> replica is >>>>>>>>> down or >>>>>>>>> at least that we are waiting for the task to complete. >>>>>>>>> Something like >>>>>>>>> that: >>>>>>>>> >>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>>>>> ... >>>>>>>>> Background task created to clean replication data >>>>>>>>> Replication data clean up may take very long time if some >>>>>>>>> replica is >>>>>>>>> unreachable >>>>>>>>> Hit CTRL+C to interrupt the wait >>>>>>>>> ^C Clean up wait interrupted >>>>>>>>> .... >>>>>>>>> [continue with del] >>>>>>>> Yup, did this in #1. >>>>>>>> >>>>>>>>> 3) (minor) When there is a cleanruv task running and you run >>>>>>>>> "ipa-replica-manage del", there is a unexpected error message >>>>>>>>> with >>>>>>>>> duplicate >>>>>>>>> task object in LDAP: >>>>>>>>> >>>>>>>>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com --force >>>>>>>>> Unable to connect to replica vm-072.idm.lab.bos.redhat.com, >>>>>>>>> forcing >>>>>>>>> removal >>>>>>>>> FAIL >>>>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': >>>>>>>>> {'desc': "Can't >>>>>>>>> contact LDAP server"} >>>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>>> >>>>>>>>> There were issues removing a connection: This entry already >>>>>>>>> exists >>>>>>>>> <<<<<<<<< >>>>>>>>> >>>>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': >>>>>>>>> {'desc': "Can't >>>>>>>>> contact LDAP server"} >>>>>>>>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: >>>>>>>>> NS record >>>>>>>>> does not >>>>>>>>> contain 'vm-072.idm.lab.bos.redhat.com.' >>>>>>>>> You may need to manually remove them from the tree >>>>>>>>> >>>>>>>>> >>>>>>>>> I think it should be enough to just catch for "entry already >>>>>>>>> exists" in >>>>>>>>> cleanallruv function, and in such case print a relevant error >>>>>>>>> message >>>>>>>>> bail out. >>>>>>>>> Thus, self.conn.checkTask(dn, dowait=True) would not be called >>>>>>>>> too. >>>>>>>> Good catch, fixed. >>>>>>>> >>>>>>>>> >>>>>>>>> 4) (minor): In make_readonly function, there is a redundant >>>>>>>>> "pass" >>>>>>>>> statement: >>>>>>>>> >>>>>>>>> + def make_readonly(self): >>>>>>>>> + """ >>>>>>>>> + Make the current replication agreement read-only. >>>>>>>>> + """ >>>>>>>>> + dn = DN(('cn', 'userRoot'), ('cn', 'ldbm database'), >>>>>>>>> + ('cn', 'plugins'), ('cn', 'config')) >>>>>>>>> + >>>>>>>>> + mod = [(ldap.MOD_REPLACE, 'nsslapd-readonly', 'on')] >>>>>>>>> + try: >>>>>>>>> + self.conn.modify_s(dn, mod) >>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>>> + # We can't make the server we're removing >>>>>>>>> read-only but >>>>>>>>> + # this isn't a show-stopper >>>>>>>>> + root_logger.debug("No permission to switch >>>>>>>>> replica to >>>>>>>>> read-only, >>>>>>>>> continuing anyway") >>>>>>>>> + pass<<<<<<<<<<<<<<< >>>>>>>> Yeah, this is one of my common mistakes. I put in a pass >>>>>>>> initially, then >>>>>>>> add logging in front of it and forget to delete the pass. Its >>>>>>>> gone now. >>>>>>>> >>>>>>>>> >>>>>>>>> 5) In clean_ruv, I think allowing a --force option to bypass the >>>>>>>>> user_input >>>>>>>>> would be helpful (at least for test automation): >>>>>>>>> >>>>>>>>> + if not ipautil.user_input("Continue to clean?", False): >>>>>>>>> + sys.exit("Aborted") >>>>>>>> Yup, added. >>>>>>>> >>>>>>>> rob >>>>>>> Slightly revised patch. I still had a window open with one >>>>>>> unsaved change. >>>>>>> >>>>>>> rob >>>>>>> >>>>>> Apparently there were two unsaved changes, one of which was lost. >>>>>> This adds in >>>>>> the 'entry already exists' fix. >>>>>> >>>>>> rob >>>>>> >>>>> Just one last thing (otherwise the patch is OK) - I don't think >>>>> this is what we >>>>> want :-) >>>>> >>>>> # ipa-replica-manage clean-ruv 8 >>>>> Clean the Replication Update Vector for >>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>> >>>>> Cleaning the wrong replica ID will cause that server to no >>>>> longer replicate so it may miss updates while the process >>>>> is running. It would need to be re-initialized to maintain >>>>> consistency. Be very careful. >>>>> Continue to clean? [no]: y<<<<<< >>>>> Aborted >>>>> >>>>> >>>>> Nor this exception, (your are checking for wrong exception): >>>>> >>>>> # ipa-replica-manage clean-ruv 8 >>>>> Clean the Replication Update Vector for >>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>> >>>>> Cleaning the wrong replica ID will cause that server to no >>>>> longer replicate so it may miss updates while the process >>>>> is running. It would need to be re-initialized to maintain >>>>> consistency. Be very careful. >>>>> Continue to clean? [no]: >>>>> unexpected error: This entry already exists >>>>> >>>>> This is the exception: >>>>> >>>>> Traceback (most recent call last): >>>>> File "/sbin/ipa-replica-manage", line 651, in >>>>> main() >>>>> File "/sbin/ipa-replica-manage", line 648, in main >>>>> clean_ruv(realm, args[1], options) >>>>> File "/sbin/ipa-replica-manage", line 373, in clean_ruv >>>>> thisrepl.cleanallruv(ruv) >>>>> File >>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>> line 1136, in cleanallruv >>>>> self.conn.addEntry(e) >>>>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", >>>>> line 503, in >>>>> addEntry >>>>> self.__handle_errors(e, arg_desc=arg_desc) >>>>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", >>>>> line 321, in >>>>> __handle_errors >>>>> raise errors.DuplicateEntry() >>>>> ipalib.errors.DuplicateEntry: This entry already exists >>>>> >>>>> Martin >>>>> >>>> >>>> On another matter, I just noticed that CLEANRUV is not proceeding >>>> if I have a >>>> winsync replica defined (and it is even up): >>>> >>>> # ipa-replica-manage list >>>> dc.ad.test: winsync<<<<<<< >>>> vm-072.idm.lab.bos.redhat.com: master >>>> vm-086.idm.lab.bos.redhat.com: master >>>> >>>> [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV >>>> Task: Waiting >>>> for all the replicas to receive all the deleted replica updates... >>>> [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV >>>> Task: Failed >>>> to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), >>>> will retry later. >>>> [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV >>>> Task: Not all >>>> replicas caught up, retrying in 10 seconds >>>> [06/Sep/2012:11:59:20 -0400] NSMMReplicationPlugin - CleanAllRUV >>>> Task: Failed >>>> to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), >>>> will retry later. >>>> [06/Sep/2012:11:59:20 -0400] NSMMReplicationPlugin - CleanAllRUV >>>> Task: Not all >>>> replicas caught up, retrying in 20 seconds >>>> [06/Sep/2012:11:59:40 -0400] NSMMReplicationPlugin - CleanAllRUV >>>> Task: Failed >>>> to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), >>>> will retry later. >>>> [06/Sep/2012:11:59:40 -0400] NSMMReplicationPlugin - CleanAllRUV >>>> Task: Not all >>>> replicas caught up, retrying in 40 seconds >>>> >>>> I don't think this is OK. Adding Rich to CC to follow on this one. >>>> >>>> Martin >>>> >>> And now the actual CC. >>> Martin >>> >> Yeah - looks like CLEANALLRUV needs to ignore windows sync agreements > Yeah, sorry not that familiar with winsync(didn't know it used the > same repl agmts). I will have to do another fix... winsync repl agreements have both objectclass: nsds5replicationagreement objectclass: nsDSWindowsReplicationAgreement >> >> _______________________________________________ >> Freeipa-devel mailing list >> Freeipa-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/freeipa-devel > From nkinder at redhat.com Thu Sep 6 18:13:03 2012 From: nkinder at redhat.com (Nathan Kinder) Date: Thu, 06 Sep 2012 11:13:03 -0700 Subject: [Freeipa-devel] Ticket #2866 - referential integrity in IPA In-Reply-To: <5048CF3D.8020408@redhat.com> References: <1345211089.3796.34.camel@priserak> <502E4E78.1090407@redhat.com> <503B9F4E.4010005@redhat.com> <503BA065.1060207@redhat.com> <5044C1F0.7080609@redhat.com> <504612DC.9060402@redhat.com> <5048CF3D.8020408@redhat.com> Message-ID: <5048E7AF.30608@redhat.com> On 09/06/2012 09:28 AM, Martin Kosek wrote: > On 09/04/2012 04:40 PM, Rich Megginson wrote: >> On 09/03/2012 08:42 AM, Martin Kosek wrote: >>> On 08/27/2012 06:29 PM, Rich Megginson wrote: >>>> On 08/27/2012 10:24 AM, Martin Kosek wrote: >>>>> On 08/17/2012 04:00 PM, Rich Megginson wrote: >>>>>> On 08/17/2012 07:44 AM, Martin Kosek wrote: >>>>>>> Hi guys, >>>>>>> >>>>>>> I am now investigating ticket #2866: >>>>>>> https://fedorahosted.org/freeipa/ticket/2866 >>>>>>> >>>>>>> And I am thinking about possible solutions for this problem. In a >>>>>>> nutshell, we do not properly check referential integrity in some IPA >>>>>>> objects where we keep one-way DN references to other objects, e.g. in >>>>>>> - managedBy attribute for a host object >>>>>>> - memberhost attribute for HBAC rule object >>>>>>> - memberuser attribute for user object >>>>>>> - memberallowcmd or memberdenycmd for SUDO command object (reported in >>>>>>> #2866) >>>>>>> ... >>>>>>> >>>>>>> Currently, I see 2 approaches to solve this: >>>>>>> 1) Add relevant checks to our ipalib plugins where problematic >>>>>>> operations with these operations are being executed (like we do for >>>>>>> selinuxusermap's seealso attribute in HBAC plugin) >>>>>>> This of course would not prevent direct LDAP deletes. >>>>>>> >>>>>>> 2) Implement a preop DS plugin that would hook to MODRDN and DELETE >>>>>>> callbacks and check that this object's DN is not referenced in other >>>>>>> objects. And if it does, it would reject such modification. Second >>>>>>> option would be to delete the attribute value with now invalid >>>>>>> reference. This would be probably more suitable for example for >>>>>>> references to user objects. >>>>>>> >>>>>>> Any comments to these possible approaches are welcome. >>>>>>> >>>>>>> Rich, do you think that as an alternative to these 2 approaches, >>>>>>> memberOf plugin could be eventually modified to do this task? >>>>>> This is very similar to the referential integrity plugin already in 389, >>>>>> except >>>>>> instead of cleaning up references to moved and deleted entries, you want >>>>>> it to >>>>>> prevent moving or deleting an entry if that entry is referenced by the >>>>>> managedby/memberhost/memberuser/memberallowcmd/memberdenycmd of some other >>>>>> entry. >>>>> I think that using or enhancing current DS referential integrity plugin >>>>> will be >>>>> the best and the most consistent way to go. >>>>> >>>>> We already use that plugin for some user attributes like "manager" or >>>>> "secretary". seeAlso is already covered by default, so for example seeAlso >>>>> attribute in SELinux usermap object referencing an HBAC rule will get removed >>>>> when relevant HBAC rule is removed (I just checked that). >>>>> >>>>>> Note that the managed entry plugin (mep) already handles this for the >>>>>> managedby >>>>>> attribute. >>>>> I assume you are referencing "mepmanagedby" and "mepmanagedentry" attributes >>>>> which then produce errors like this one: >>>>> >>>>> # ipa netgroup-del foo >>>>> ipa: ERROR: Server is unwilling to perform: Deleting a managed entry is not >>>>> allowed. It needs to be manually unlinked first. >>>>> >>>>> managedBy attribute used by hosts objects I had in mind seems to not be >>>>> covered. >>>>> >>>>> But you are right, this is pretty much what I wanted. Though in case of MEP, >>>>> there is a link in both referenced objects, but in our case, we have just the >>>>> one-way link. >>>>> >>>>>> Are you already using the memberof plugin for >>>>>> memberhost/memberuser/memberallowcmd/memberdenycmd? >>>>>> >>>>>> This doesn't seem like a job for memberof, this seems like more of a new >>>>>> check >>>>>> for the referential integrity plugin. >>>>>> >>>>> I am now considering if current move/delete clean up already present in >>>>> Referential Integrity plugin would not be sufficient for us. >>>>> >>>>> Rich, please correct me if I am wrong, but in that case, we would just need to >>>>> add relevant attribute names >>>>> (memberhost/memberuser/memberallowcmd/memberdenycmd...) to Referential >>>>> Integrity plugin configuration as nsslapd-pluginarg7, nsslapd-pluginarg8, ... >>>>> I wonder if there would be some performance issues if we add attributes to the >>>>> list this way. >>>> No, not if they are indexed for presence and equality. >>>> >>> Hello Rich, >>> I am back to investigate this ticket. In order to be able to deliver some >>> working solution to IPA 3.0, I plan to take advantage of current Referential >>> Integrity Plugin to clean up dangling references. >>> >>> This is the plan I plan to take: >>> 1) Add pres,eq indexes for all un-indexed attributes that we want to check: >>> sourcehost >>> memberservice >>> managedby >>> memberallowcmd >>> memberdenycmd >>> ipasudorunas >>> ipasudorunasgroup >> ok >>> 2) Add missing "pres" index for attributes we want to check, but only have eq >>> index: >>> manager >>> secretary >>> memberuser >>> memberhost >>> >>> I assume this step is also needed in order to keep the server performance. >> yes >>> 3) Add all these attributes do Referential Integrity Plugin attribute list of >>> not already >> ok >>> 4) Also add Index task (nsIndexAttribute) for all these new indexes so that >>> they are created during IPA server upgrade. >> ok >>> Is this procedure OK DS-wise? >> Yes >>> I also have question regarding the following note in RHDS doc chapter 3.6. >>> Maintaining Referential Integrity: >>> >>> "The Referential Integrity Plug-in should only be enabled on one supplier >>> replica in a multi-master replication environment to avoid conflict resolution >>> loops..." >>> >>> Currently, we enable this plugin on all IPA replicas. Is this something we need >>> to be concerned about and fix ASAP (before we do all this RefInt effort)? >> Mark/Nathan - I know you guys have looked at this issue. > Hello guys, > > any update on that? Just checking before I implement this extended referential > check in IPA. The last I looked at this was a few years ago. From what I remember, I wasn't ever convinced that the documentation was correct, but I couldn't prove that it was incorrect either. I will dig some more to see if I can find any of my notes from that investigation. -NGK > > Thanks, > Martin From mkosek at redhat.com Thu Sep 6 18:23:49 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 06 Sep 2012 20:23:49 +0200 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <5048D280.2050607@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> <50479491.1060200@redhat.com> <50488DD0.1080507@redhat.com> <5048BEF8.5000401@redhat.com> <5048C46A.8020207@redhat.com> <5048C754.2030708@redhat.com> <5048C9BE.1050005@redhat.com> <5048CA9F.7060607@redhat.com> <5048CAD0.2070508@redhat.com> <5048CB9E.4090301@redhat.com> <5048CEE5.80704@redhat.com> <5048D280.2050607@redhat.com> Message-ID: <1346955829.2191.0.camel@priserak> On Thu, 2012-09-06 at 12:42 -0400, Mark Reynolds wrote: > > On 09/06/2012 12:27 PM, Martin Kosek wrote: > > On 09/06/2012 06:13 PM, Rich Megginson wrote: > >> On 09/06/2012 10:09 AM, Martin Kosek wrote: > >>> On 09/06/2012 06:09 PM, Martin Kosek wrote: > >>>> On 09/06/2012 06:05 PM, Martin Kosek wrote: > >>>>> On 09/06/2012 05:55 PM, Rob Crittenden wrote: > >>>>>> Rob Crittenden wrote: > >>>>>>> Rob Crittenden wrote: > >>>>>>>> Martin Kosek wrote: > >>>>>>>>> On 09/05/2012 08:06 PM, Rob Crittenden wrote: > >>>>>>>>>> Rob Crittenden wrote: > >>>>>>>>>>> Martin Kosek wrote: > >>>>>>>>>>>> On 07/05/2012 08:39 PM, Rob Crittenden wrote: > >>>>>>>>>>>>> Martin Kosek wrote: > >>>>>>>>>>>>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: > >>>>>>>>>>>>>>> Deleting a replica can leave a replication vector (RUV) on the > >>>>>>>>>>>>>>> other servers. > >>>>>>>>>>>>>>> This can confuse things if the replica is re-added, and it also > >>>>>>>>>>>>>>> causes the > >>>>>>>>>>>>>>> server to calculate changes against a server that may no longer > >>>>>>>>>>>>>>> exist. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> 389-ds-base provides a new task that self-propogates itself to all > >>>>>>>>>>>>>>> available > >>>>>>>>>>>>>>> replicas to clean this RUV data. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> This patch will create this task at deletion time to hopefully > >>>>>>>>>>>>>>> clean things up. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> It isn't perfect. If any replica is down or unavailable at the > >>>>>>>>>>>>>>> time > >>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>> cleanruv task fires, and then comes back up, the old RUV data > >>>>>>>>>>>>>>> may be > >>>>>>>>>>>>>>> re-propogated around. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> To make things easier in this case I've added two new commands to > >>>>>>>>>>>>>>> ipa-replica-manage. The first lists the replication ids of all the > >>>>>>>>>>>>>>> servers we > >>>>>>>>>>>>>>> have a RUV for. Using this you can call clean_ruv with the > >>>>>>>>>>>>>>> replication id of a > >>>>>>>>>>>>>>> server that no longer exists to try the cleanallruv step again. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> This is quite dangerous though. If you run cleanruv against a > >>>>>>>>>>>>>>> replica id that > >>>>>>>>>>>>>>> does exist it can cause a loss of data. I believe I've put in > >>>>>>>>>>>>>>> enough scary > >>>>>>>>>>>>>>> warnings about this. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> rob > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> Good work there, this should make cleaning RUVs much easier than > >>>>>>>>>>>>>> with the > >>>>>>>>>>>>>> previous version. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> This is what I found during review: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 1) list_ruv and clean_ruv command help in man is quite lost. I > >>>>>>>>>>>>>> think > >>>>>>>>>>>>>> it would > >>>>>>>>>>>>>> help if we for example have all info for commands indented. This > >>>>>>>>>>>>>> way > >>>>>>>>>>>>>> user could > >>>>>>>>>>>>>> simply over-look the new commands in the man page. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 2) I would rename new commands to clean-ruv and list-ruv to make > >>>>>>>>>>>>>> them > >>>>>>>>>>>>>> consistent with the rest of the commands (re-initialize, > >>>>>>>>>>>>>> force-sync). > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 3) It would be nice to be able to run clean_ruv command in an > >>>>>>>>>>>>>> unattended way > >>>>>>>>>>>>>> (for better testing), i.e. respect --force option as we already > >>>>>>>>>>>>>> do for > >>>>>>>>>>>>>> ipa-replica-manage del. This fix would aid test automation in the > >>>>>>>>>>>>>> future. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 4) (minor) The new question (and the del too) does not react too > >>>>>>>>>>>>>> well for > >>>>>>>>>>>>>> CTRL+D: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> # ipa-replica-manage clean_ruv 3 --force > >>>>>>>>>>>>>> Clean the Replication Update Vector for > >>>>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no > >>>>>>>>>>>>>> longer replicate so it may miss updates while the process > >>>>>>>>>>>>>> is running. It would need to be re-initialized to maintain > >>>>>>>>>>>>>> consistency. Be very careful. > >>>>>>>>>>>>>> Continue to clean? [no]: unexpected error: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 5) Help for clean_ruv command without a required parameter is quite > >>>>>>>>>>>>>> confusing > >>>>>>>>>>>>>> as it reports that command is wrong and not the parameter: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> # ipa-replica-manage clean_ruv > >>>>>>>>>>>>>> Usage: ipa-replica-manage [options] > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> ipa-replica-manage: error: must provide a command [clean_ruv | > >>>>>>>>>>>>>> force-sync | > >>>>>>>>>>>>>> disconnect | connect | del | re-initialize | list | list_ruv] > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> It seems you just forgot to specify the error message in the > >>>>>>>>>>>>>> command > >>>>>>>>>>>>>> definition > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 6) When the remote replica is down, the clean_ruv command fails > >>>>>>>>>>>>>> with an > >>>>>>>>>>>>>> unexpected error: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 > >>>>>>>>>>>>>> Clean the Replication Update Vector for > >>>>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no > >>>>>>>>>>>>>> longer replicate so it may miss updates while the process > >>>>>>>>>>>>>> is running. It would need to be re-initialized to maintain > >>>>>>>>>>>>>> consistency. Be very careful. > >>>>>>>>>>>>>> Continue to clean? [no]: y > >>>>>>>>>>>>>> unexpected error: {'desc': 'Operations error'} > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: > >>>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - > >>>>>>>>>>>>>> cleanAllRUV_task: failed > >>>>>>>>>>>>>> to connect to repl agreement connection > >>>>>>>>>>>>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> tree,cn=config), error 105 > >>>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - > >>>>>>>>>>>>>> cleanAllRUV_task: replica > >>>>>>>>>>>>>> (cn=meTovm-055.idm.lab. > >>>>>>>>>>>>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> tree, cn=config) has not been cleaned. You will need to rerun > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>> CLEANALLRUV task on this replica. > >>>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - > >>>>>>>>>>>>>> cleanAllRUV_task: Task > >>>>>>>>>>>>>> failed (1) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> In this case I think we should inform user that the command failed, > >>>>>>>>>>>>>> possibly > >>>>>>>>>>>>>> because of disconnected replicas and that they could enable the > >>>>>>>>>>>>>> replicas and > >>>>>>>>>>>>>> try again. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 7) (minor) "pass" is now redundant in replication.py: > >>>>>>>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: > >>>>>>>>>>>>>> + # We can't make the server we're removing read-only > >>>>>>>>>>>>>> but > >>>>>>>>>>>>>> + # this isn't a show-stopper > >>>>>>>>>>>>>> + root_logger.debug("No permission to switch replica to > >>>>>>>>>>>>>> read-only, > >>>>>>>>>>>>>> continuing anyway") > >>>>>>>>>>>>>> + pass > >>>>>>>>>>>>>> > >>>>>>>>>>>>> I think this addresses everything. > >>>>>>>>>>>>> > >>>>>>>>>>>>> rob > >>>>>>>>>>>> Thanks, almost there! I just found one more issue which needs to be > >>>>>>>>>>>> fixed > >>>>>>>>>>>> before we push: > >>>>>>>>>>>> > >>>>>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force > >>>>>>>>>>>> Directory Manager password: > >>>>>>>>>>>> > >>>>>>>>>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing > >>>>>>>>>>>> removal > >>>>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': > >>>>>>>>>>>> "Can't > >>>>>>>>>>>> contact LDAP server"} > >>>>>>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' > >>>>>>>>>>>> > >>>>>>>>>>>> There were issues removing a connection: %d format: a number is > >>>>>>>>>>>> required, not str > >>>>>>>>>>>> > >>>>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': > >>>>>>>>>>>> "Can't > >>>>>>>>>>>> contact LDAP server"} > >>>>>>>>>>>> > >>>>>>>>>>>> This is a traceback I retrieved: > >>>>>>>>>>>> Traceback (most recent call last): > >>>>>>>>>>>> File "/sbin/ipa-replica-manage", line 425, in del_master > >>>>>>>>>>>> del_link(realm, r, hostname, options.dirman_passwd, force=True) > >>>>>>>>>>>> File "/sbin/ipa-replica-manage", line 271, in del_link > >>>>>>>>>>>> repl1.cleanallruv(replica_id) > >>>>>>>>>>>> File > >>>>>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", > >>>>>>>>>>>> line 1094, in cleanallruv > >>>>>>>>>>>> root_logger.debug("Creating CLEANALLRUV task for replica id > >>>>>>>>>>>> %d" % > >>>>>>>>>>>> replicaId) > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> The problem here is that you don't convert replica_id to int in this > >>>>>>>>>>>> part: > >>>>>>>>>>>> + replica_id = None > >>>>>>>>>>>> + if repl2: > >>>>>>>>>>>> + replica_id = repl2._get_replica_id(repl2.conn, None) > >>>>>>>>>>>> + else: > >>>>>>>>>>>> + servers = get_ruv(realm, replica1, dirman_passwd) > >>>>>>>>>>>> + for (netloc, rid) in servers: > >>>>>>>>>>>> + if netloc.startswith(replica2): > >>>>>>>>>>>> + replica_id = rid > >>>>>>>>>>>> + break > >>>>>>>>>>>> > >>>>>>>>>>>> Martin > >>>>>>>>>>>> > >>>>>>>>>>> Updated patch using new mechanism in 389-ds-base. This should more > >>>>>>>>>>> thoroughly clean out RUV data when a replica is being deleted, and > >>>>>>>>>>> provide for a way to delete RUV data afterwards too if necessary. > >>>>>>>>>>> > >>>>>>>>>>> rob > >>>>>>>>>> Rebased patch > >>>>>>>>>> > >>>>>>>>>> rob > >>>>>>>>>> > >>>>>>>>> 0) As I wrote in a review for your patch 1041, changelog entry slipped > >>>>>>>>> elsewhere. > >>>>>>>>> > >>>>>>>>> 1) The following KeyboardInterrupt except class looks suspicious. I > >>>>>>>>> know why > >>>>>>>>> you have it there, but since it is generally a bad thing to do, some > >>>>>>>>> comment > >>>>>>>>> why it is needed would be useful. > >>>>>>>>> > >>>>>>>>> @@ -256,6 +263,17 @@ def del_link(realm, replica1, replica2, > >>>>>>>>> dirman_passwd, > >>>>>>>>> force=False): > >>>>>>>>> repl1.delete_agreement(replica2) > >>>>>>>>> repl1.delete_referral(replica2) > >>>>>>>>> > >>>>>>>>> + if type1 == replication.IPA_REPLICA: > >>>>>>>>> + if repl2: > >>>>>>>>> + ruv = repl2._get_replica_id(repl2.conn, None) > >>>>>>>>> + else: > >>>>>>>>> + ruv = get_ruv_by_host(realm, replica1, replica2, > >>>>>>>>> dirman_passwd) > >>>>>>>>> + > >>>>>>>>> + try: > >>>>>>>>> + repl1.cleanallruv(ruv) > >>>>>>>>> + except KeyboardInterrupt: > >>>>>>>>> + pass > >>>>>>>>> + > >>>>>>>>> > >>>>>>>>> Maybe you just wanted to do some cleanup and then "raise" again? > >>>>>>>> No, it is there because it is safe to break out of it. The task will > >>>>>>>> continue to run. I added some verbiage. > >>>>>>>> > >>>>>>>>> 2) This is related to 1), but when some replica is down, > >>>>>>>>> "ipa-replica-manage > >>>>>>>>> del" may wait indefinitely when some remote replica is down, right? > >>>>>>>>> > >>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com > >>>>>>>>> Deleting a master is irreversible. > >>>>>>>>> To reconnect to the remote master you will need to prepare a new > >>>>>>>>> replica file > >>>>>>>>> and re-install. > >>>>>>>>> Continue to delete? [no]: y > >>>>>>>>> ipa: INFO: Setting agreement > >>>>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> tree,cn=config schedule to 2358-2359 0 to force synch > >>>>>>>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement > >>>>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> tree,cn=config > >>>>>>>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica > >>>>>>>>> acquired > >>>>>>>>> successfully: Incremental update succeeded: start: 0: end: 0 > >>>>>>>>> Background task created to clean replication data > >>>>>>>>> > >>>>>>>>> ... after about a minute I hit CTRL+C > >>>>>>>>> > >>>>>>>>> ^CDeleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to > >>>>>>>>> 'vm-055.idm.lab.bos.redhat.com' > >>>>>>>>> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: NS record > >>>>>>>>> does not > >>>>>>>>> contain 'vm-055.idm.lab.bos.redhat.com.' > >>>>>>>>> You may need to manually remove them from the tree > >>>>>>>>> > >>>>>>>>> I think it would be better to inform user that some remote replica is > >>>>>>>>> down or > >>>>>>>>> at least that we are waiting for the task to complete. Something like > >>>>>>>>> that: > >>>>>>>>> > >>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com > >>>>>>>>> ... > >>>>>>>>> Background task created to clean replication data > >>>>>>>>> Replication data clean up may take very long time if some replica is > >>>>>>>>> unreachable > >>>>>>>>> Hit CTRL+C to interrupt the wait > >>>>>>>>> ^C Clean up wait interrupted > >>>>>>>>> .... > >>>>>>>>> [continue with del] > >>>>>>>> Yup, did this in #1. > >>>>>>>> > >>>>>>>>> 3) (minor) When there is a cleanruv task running and you run > >>>>>>>>> "ipa-replica-manage del", there is a unexpected error message with > >>>>>>>>> duplicate > >>>>>>>>> task object in LDAP: > >>>>>>>>> > >>>>>>>>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com --force > >>>>>>>>> Unable to connect to replica vm-072.idm.lab.bos.redhat.com, forcing > >>>>>>>>> removal > >>>>>>>>> FAIL > >>>>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't > >>>>>>>>> contact LDAP server"} > >>>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' > >>>>>>>>> > >>>>>>>>> There were issues removing a connection: This entry already exists > >>>>>>>>> <<<<<<<<< > >>>>>>>>> > >>>>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't > >>>>>>>>> contact LDAP server"} > >>>>>>>>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record > >>>>>>>>> does not > >>>>>>>>> contain 'vm-072.idm.lab.bos.redhat.com.' > >>>>>>>>> You may need to manually remove them from the tree > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> I think it should be enough to just catch for "entry already exists" in > >>>>>>>>> cleanallruv function, and in such case print a relevant error message > >>>>>>>>> bail out. > >>>>>>>>> Thus, self.conn.checkTask(dn, dowait=True) would not be called too. > >>>>>>>> Good catch, fixed. > >>>>>>>> > >>>>>>>>> 4) (minor): In make_readonly function, there is a redundant "pass" > >>>>>>>>> statement: > >>>>>>>>> > >>>>>>>>> + def make_readonly(self): > >>>>>>>>> + """ > >>>>>>>>> + Make the current replication agreement read-only. > >>>>>>>>> + """ > >>>>>>>>> + dn = DN(('cn', 'userRoot'), ('cn', 'ldbm database'), > >>>>>>>>> + ('cn', 'plugins'), ('cn', 'config')) > >>>>>>>>> + > >>>>>>>>> + mod = [(ldap.MOD_REPLACE, 'nsslapd-readonly', 'on')] > >>>>>>>>> + try: > >>>>>>>>> + self.conn.modify_s(dn, mod) > >>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: > >>>>>>>>> + # We can't make the server we're removing read-only but > >>>>>>>>> + # this isn't a show-stopper > >>>>>>>>> + root_logger.debug("No permission to switch replica to > >>>>>>>>> read-only, > >>>>>>>>> continuing anyway") > >>>>>>>>> + pass<<<<<<<<<<<<<<< > >>>>>>>> Yeah, this is one of my common mistakes. I put in a pass initially, then > >>>>>>>> add logging in front of it and forget to delete the pass. Its gone now. > >>>>>>>> > >>>>>>>>> 5) In clean_ruv, I think allowing a --force option to bypass the > >>>>>>>>> user_input > >>>>>>>>> would be helpful (at least for test automation): > >>>>>>>>> > >>>>>>>>> + if not ipautil.user_input("Continue to clean?", False): > >>>>>>>>> + sys.exit("Aborted") > >>>>>>>> Yup, added. > >>>>>>>> > >>>>>>>> rob > >>>>>>> Slightly revised patch. I still had a window open with one unsaved change. > >>>>>>> > >>>>>>> rob > >>>>>>> > >>>>>> Apparently there were two unsaved changes, one of which was lost. This > >>>>>> adds in > >>>>>> the 'entry already exists' fix. > >>>>>> > >>>>>> rob > >>>>>> > >>>>> Just one last thing (otherwise the patch is OK) - I don't think this is > >>>>> what we > >>>>> want :-) > >>>>> > >>>>> # ipa-replica-manage clean-ruv 8 > >>>>> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 > >>>>> > >>>>> Cleaning the wrong replica ID will cause that server to no > >>>>> longer replicate so it may miss updates while the process > >>>>> is running. It would need to be re-initialized to maintain > >>>>> consistency. Be very careful. > >>>>> Continue to clean? [no]: y<<<<<< > >>>>> Aborted > >>>>> > >>>>> > >>>>> Nor this exception, (your are checking for wrong exception): > >>>>> > >>>>> # ipa-replica-manage clean-ruv 8 > >>>>> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 > >>>>> > >>>>> Cleaning the wrong replica ID will cause that server to no > >>>>> longer replicate so it may miss updates while the process > >>>>> is running. It would need to be re-initialized to maintain > >>>>> consistency. Be very careful. > >>>>> Continue to clean? [no]: > >>>>> unexpected error: This entry already exists > >>>>> > >>>>> This is the exception: > >>>>> > >>>>> Traceback (most recent call last): > >>>>> File "/sbin/ipa-replica-manage", line 651, in > >>>>> main() > >>>>> File "/sbin/ipa-replica-manage", line 648, in main > >>>>> clean_ruv(realm, args[1], options) > >>>>> File "/sbin/ipa-replica-manage", line 373, in clean_ruv > >>>>> thisrepl.cleanallruv(ruv) > >>>>> File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", > >>>>> line 1136, in cleanallruv > >>>>> self.conn.addEntry(e) > >>>>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 503, in > >>>>> addEntry > >>>>> self.__handle_errors(e, arg_desc=arg_desc) > >>>>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 321, in > >>>>> __handle_errors > >>>>> raise errors.DuplicateEntry() > >>>>> ipalib.errors.DuplicateEntry: This entry already exists > >>>>> > >>>>> Martin > >>>>> > >>>> On another matter, I just noticed that CLEANRUV is not proceeding if I have a > >>>> winsync replica defined (and it is even up): > >>>> > >>>> # ipa-replica-manage list > >>>> dc.ad.test: winsync<<<<<<< > >>>> vm-072.idm.lab.bos.redhat.com: master > >>>> vm-086.idm.lab.bos.redhat.com: master > >>>> > >>>> [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Waiting > >>>> for all the replicas to receive all the deleted replica updates... > >>>> [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Failed > >>>> to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), will retry > >>>> later. > >>>> [06/Sep/2012:11:59:10 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all > >>>> replicas caught up, retrying in 10 seconds > >>>> [06/Sep/2012:11:59:20 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Failed > >>>> to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), will retry > >>>> later. > >>>> [06/Sep/2012:11:59:20 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all > >>>> replicas caught up, retrying in 20 seconds > >>>> [06/Sep/2012:11:59:40 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Failed > >>>> to contact agmt (agmt="cn=meTodc.ad.test" (dc:389)) error (10), will retry > >>>> later. > >>>> [06/Sep/2012:11:59:40 -0400] NSMMReplicationPlugin - CleanAllRUV Task: Not all > >>>> replicas caught up, retrying in 40 seconds > >>>> > >>>> I don't think this is OK. Adding Rich to CC to follow on this one. > >>>> > >>>> Martin > >>>> > >>> And now the actual CC. > >>> Martin > >>> > >> Yeah - looks like CLEANALLRUV needs to ignore windows sync agreements > > Have you created a 389 ticket for that or would you like me to do that? > If you could, that would be great. https://fedorahosted.org/389/ticket/450 Martin From mkosek at redhat.com Thu Sep 6 18:34:45 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 06 Sep 2012 20:34:45 +0200 Subject: [Freeipa-devel] [PATCH] 303 Add range safety check for range_mod and range_del In-Reply-To: <20120906155519.GA30746@localhost.localdomain> References: <50476C25.9000009@redhat.com> <20120906155519.GA30746@localhost.localdomain> Message-ID: <1346956485.2191.3.camel@priserak> On Thu, 2012-09-06 at 17:55 +0200, Sumit Bose wrote: > On Wed, Sep 05, 2012 at 05:13:41PM +0200, Martin Kosek wrote: > > range_mod and range_del command could easily create objects with > > ID which is suddenly out of specified range. This could cause issues > > in trust scenarios where range objects are used for computation of > > remote IDs. > > > > Add validator for both commands to check if there is any object with > > ID in the range which would become out-of-range as a pre_callback. > > Also add unit tests testing this new validator. > > > > https://fedorahosted.org/freeipa/ticket/2919 > > ACK > > bye, > Sumit > Thanks for the review. Pushed to master, ipa-3-0. Martin From rcritten at redhat.com Thu Sep 6 21:17:45 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 06 Sep 2012 17:17:45 -0400 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <5048C9BE.1050005@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> <50479491.1060200@redhat.com> <50488DD0.1080507@redhat.com> <5048BEF8.5000401@redhat.com> <5048C46A.8020207@redhat.com> <5048C754.2030708@redhat.com> <5048C9BE.1050005@redhat.com> Message-ID: <504912F9.1020907@redhat.com> Martin Kosek wrote: > On 09/06/2012 05:55 PM, Rob Crittenden wrote: >> Rob Crittenden wrote: >>> Rob Crittenden wrote: >>>> Martin Kosek wrote: >>>>> On 09/05/2012 08:06 PM, Rob Crittenden wrote: >>>>>> Rob Crittenden wrote: >>>>>>> Martin Kosek wrote: >>>>>>>> On 07/05/2012 08:39 PM, Rob Crittenden wrote: >>>>>>>>> Martin Kosek wrote: >>>>>>>>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: >>>>>>>>>>> Deleting a replica can leave a replication vector (RUV) on the >>>>>>>>>>> other servers. >>>>>>>>>>> This can confuse things if the replica is re-added, and it also >>>>>>>>>>> causes the >>>>>>>>>>> server to calculate changes against a server that may no longer >>>>>>>>>>> exist. >>>>>>>>>>> >>>>>>>>>>> 389-ds-base provides a new task that self-propogates itself to all >>>>>>>>>>> available >>>>>>>>>>> replicas to clean this RUV data. >>>>>>>>>>> >>>>>>>>>>> This patch will create this task at deletion time to hopefully >>>>>>>>>>> clean things up. >>>>>>>>>>> >>>>>>>>>>> It isn't perfect. If any replica is down or unavailable at the >>>>>>>>>>> time >>>>>>>>>>> the >>>>>>>>>>> cleanruv task fires, and then comes back up, the old RUV data >>>>>>>>>>> may be >>>>>>>>>>> re-propogated around. >>>>>>>>>>> >>>>>>>>>>> To make things easier in this case I've added two new commands to >>>>>>>>>>> ipa-replica-manage. The first lists the replication ids of all the >>>>>>>>>>> servers we >>>>>>>>>>> have a RUV for. Using this you can call clean_ruv with the >>>>>>>>>>> replication id of a >>>>>>>>>>> server that no longer exists to try the cleanallruv step again. >>>>>>>>>>> >>>>>>>>>>> This is quite dangerous though. If you run cleanruv against a >>>>>>>>>>> replica id that >>>>>>>>>>> does exist it can cause a loss of data. I believe I've put in >>>>>>>>>>> enough scary >>>>>>>>>>> warnings about this. >>>>>>>>>>> >>>>>>>>>>> rob >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Good work there, this should make cleaning RUVs much easier than >>>>>>>>>> with the >>>>>>>>>> previous version. >>>>>>>>>> >>>>>>>>>> This is what I found during review: >>>>>>>>>> >>>>>>>>>> 1) list_ruv and clean_ruv command help in man is quite lost. I >>>>>>>>>> think >>>>>>>>>> it would >>>>>>>>>> help if we for example have all info for commands indented. This >>>>>>>>>> way >>>>>>>>>> user could >>>>>>>>>> simply over-look the new commands in the man page. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2) I would rename new commands to clean-ruv and list-ruv to make >>>>>>>>>> them >>>>>>>>>> consistent with the rest of the commands (re-initialize, >>>>>>>>>> force-sync). >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 3) It would be nice to be able to run clean_ruv command in an >>>>>>>>>> unattended way >>>>>>>>>> (for better testing), i.e. respect --force option as we already >>>>>>>>>> do for >>>>>>>>>> ipa-replica-manage del. This fix would aid test automation in the >>>>>>>>>> future. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 4) (minor) The new question (and the del too) does not react too >>>>>>>>>> well for >>>>>>>>>> CTRL+D: >>>>>>>>>> >>>>>>>>>> # ipa-replica-manage clean_ruv 3 --force >>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>> >>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>> consistency. Be very careful. >>>>>>>>>> Continue to clean? [no]: unexpected error: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 5) Help for clean_ruv command without a required parameter is quite >>>>>>>>>> confusing >>>>>>>>>> as it reports that command is wrong and not the parameter: >>>>>>>>>> >>>>>>>>>> # ipa-replica-manage clean_ruv >>>>>>>>>> Usage: ipa-replica-manage [options] >>>>>>>>>> >>>>>>>>>> ipa-replica-manage: error: must provide a command [clean_ruv | >>>>>>>>>> force-sync | >>>>>>>>>> disconnect | connect | del | re-initialize | list | list_ruv] >>>>>>>>>> >>>>>>>>>> It seems you just forgot to specify the error message in the >>>>>>>>>> command >>>>>>>>>> definition >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 6) When the remote replica is down, the clean_ruv command fails >>>>>>>>>> with an >>>>>>>>>> unexpected error: >>>>>>>>>> >>>>>>>>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 >>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>> >>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>> consistency. Be very careful. >>>>>>>>>> Continue to clean? [no]: y >>>>>>>>>> unexpected error: {'desc': 'Operations error'} >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: >>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>> cleanAllRUV_task: failed >>>>>>>>>> to connect to repl agreement connection >>>>>>>>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, >>>>>>>>>> >>>>>>>>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> tree,cn=config), error 105 >>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>> cleanAllRUV_task: replica >>>>>>>>>> (cn=meTovm-055.idm.lab. >>>>>>>>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> tree, cn=config) has not been cleaned. You will need to rerun >>>>>>>>>> the >>>>>>>>>> CLEANALLRUV task on this replica. >>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>> cleanAllRUV_task: Task >>>>>>>>>> failed (1) >>>>>>>>>> >>>>>>>>>> In this case I think we should inform user that the command failed, >>>>>>>>>> possibly >>>>>>>>>> because of disconnected replicas and that they could enable the >>>>>>>>>> replicas and >>>>>>>>>> try again. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 7) (minor) "pass" is now redundant in replication.py: >>>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>>>> + # We can't make the server we're removing read-only >>>>>>>>>> but >>>>>>>>>> + # this isn't a show-stopper >>>>>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>>>>> read-only, >>>>>>>>>> continuing anyway") >>>>>>>>>> + pass >>>>>>>>>> >>>>>>>>> >>>>>>>>> I think this addresses everything. >>>>>>>>> >>>>>>>>> rob >>>>>>>> >>>>>>>> Thanks, almost there! I just found one more issue which needs to be >>>>>>>> fixed >>>>>>>> before we push: >>>>>>>> >>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>>>>>> Directory Manager password: >>>>>>>> >>>>>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing >>>>>>>> removal >>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>> "Can't >>>>>>>> contact LDAP server"} >>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>> >>>>>>>> There were issues removing a connection: %d format: a number is >>>>>>>> required, not str >>>>>>>> >>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>> "Can't >>>>>>>> contact LDAP server"} >>>>>>>> >>>>>>>> This is a traceback I retrieved: >>>>>>>> Traceback (most recent call last): >>>>>>>> File "/sbin/ipa-replica-manage", line 425, in del_master >>>>>>>> del_link(realm, r, hostname, options.dirman_passwd, force=True) >>>>>>>> File "/sbin/ipa-replica-manage", line 271, in del_link >>>>>>>> repl1.cleanallruv(replica_id) >>>>>>>> File >>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>>>>> line 1094, in cleanallruv >>>>>>>> root_logger.debug("Creating CLEANALLRUV task for replica id >>>>>>>> %d" % >>>>>>>> replicaId) >>>>>>>> >>>>>>>> >>>>>>>> The problem here is that you don't convert replica_id to int in this >>>>>>>> part: >>>>>>>> + replica_id = None >>>>>>>> + if repl2: >>>>>>>> + replica_id = repl2._get_replica_id(repl2.conn, None) >>>>>>>> + else: >>>>>>>> + servers = get_ruv(realm, replica1, dirman_passwd) >>>>>>>> + for (netloc, rid) in servers: >>>>>>>> + if netloc.startswith(replica2): >>>>>>>> + replica_id = rid >>>>>>>> + break >>>>>>>> >>>>>>>> Martin >>>>>>>> >>>>>>> >>>>>>> Updated patch using new mechanism in 389-ds-base. This should more >>>>>>> thoroughly clean out RUV data when a replica is being deleted, and >>>>>>> provide for a way to delete RUV data afterwards too if necessary. >>>>>>> >>>>>>> rob >>>>>> >>>>>> Rebased patch >>>>>> >>>>>> rob >>>>>> >>>>> >>>>> 0) As I wrote in a review for your patch 1041, changelog entry slipped >>>>> elsewhere. >>>>> >>>>> 1) The following KeyboardInterrupt except class looks suspicious. I >>>>> know why >>>>> you have it there, but since it is generally a bad thing to do, some >>>>> comment >>>>> why it is needed would be useful. >>>>> >>>>> @@ -256,6 +263,17 @@ def del_link(realm, replica1, replica2, >>>>> dirman_passwd, >>>>> force=False): >>>>> repl1.delete_agreement(replica2) >>>>> repl1.delete_referral(replica2) >>>>> >>>>> + if type1 == replication.IPA_REPLICA: >>>>> + if repl2: >>>>> + ruv = repl2._get_replica_id(repl2.conn, None) >>>>> + else: >>>>> + ruv = get_ruv_by_host(realm, replica1, replica2, >>>>> dirman_passwd) >>>>> + >>>>> + try: >>>>> + repl1.cleanallruv(ruv) >>>>> + except KeyboardInterrupt: >>>>> + pass >>>>> + >>>>> >>>>> Maybe you just wanted to do some cleanup and then "raise" again? >>>> >>>> No, it is there because it is safe to break out of it. The task will >>>> continue to run. I added some verbiage. >>>> >>>>> >>>>> 2) This is related to 1), but when some replica is down, >>>>> "ipa-replica-manage >>>>> del" may wait indefinitely when some remote replica is down, right? >>>>> >>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>> Deleting a master is irreversible. >>>>> To reconnect to the remote master you will need to prepare a new >>>>> replica file >>>>> and re-install. >>>>> Continue to delete? [no]: y >>>>> ipa: INFO: Setting agreement >>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>> >>>>> >>>>> >>>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>> >>>>> >>>>> >>>>> tree,cn=config >>>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica >>>>> acquired >>>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>>> Background task created to clean replication data >>>>> >>>>> ... after about a minute I hit CTRL+C >>>>> >>>>> ^CDeleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to >>>>> 'vm-055.idm.lab.bos.redhat.com' >>>>> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: NS record >>>>> does not >>>>> contain 'vm-055.idm.lab.bos.redhat.com.' >>>>> You may need to manually remove them from the tree >>>>> >>>>> I think it would be better to inform user that some remote replica is >>>>> down or >>>>> at least that we are waiting for the task to complete. Something like >>>>> that: >>>>> >>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>> ... >>>>> Background task created to clean replication data >>>>> Replication data clean up may take very long time if some replica is >>>>> unreachable >>>>> Hit CTRL+C to interrupt the wait >>>>> ^C Clean up wait interrupted >>>>> .... >>>>> [continue with del] >>>> >>>> Yup, did this in #1. >>>> >>>>> >>>>> 3) (minor) When there is a cleanruv task running and you run >>>>> "ipa-replica-manage del", there is a unexpected error message with >>>>> duplicate >>>>> task object in LDAP: >>>>> >>>>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com --force >>>>> Unable to connect to replica vm-072.idm.lab.bos.redhat.com, forcing >>>>> removal >>>>> FAIL >>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>> contact LDAP server"} >>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>> >>>>> There were issues removing a connection: This entry already exists >>>>> <<<<<<<<< >>>>> >>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>> contact LDAP server"} >>>>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record >>>>> does not >>>>> contain 'vm-072.idm.lab.bos.redhat.com.' >>>>> You may need to manually remove them from the tree >>>>> >>>>> >>>>> I think it should be enough to just catch for "entry already exists" in >>>>> cleanallruv function, and in such case print a relevant error message >>>>> bail out. >>>>> Thus, self.conn.checkTask(dn, dowait=True) would not be called too. >>>> >>>> Good catch, fixed. >>>> >>>>> >>>>> >>>>> 4) (minor): In make_readonly function, there is a redundant "pass" >>>>> statement: >>>>> >>>>> + def make_readonly(self): >>>>> + """ >>>>> + Make the current replication agreement read-only. >>>>> + """ >>>>> + dn = DN(('cn', 'userRoot'), ('cn', 'ldbm database'), >>>>> + ('cn', 'plugins'), ('cn', 'config')) >>>>> + >>>>> + mod = [(ldap.MOD_REPLACE, 'nsslapd-readonly', 'on')] >>>>> + try: >>>>> + self.conn.modify_s(dn, mod) >>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>> + # We can't make the server we're removing read-only but >>>>> + # this isn't a show-stopper >>>>> + root_logger.debug("No permission to switch replica to >>>>> read-only, >>>>> continuing anyway") >>>>> + pass <<<<<<<<<<<<<<< >>>> >>>> Yeah, this is one of my common mistakes. I put in a pass initially, then >>>> add logging in front of it and forget to delete the pass. Its gone now. >>>> >>>>> >>>>> >>>>> 5) In clean_ruv, I think allowing a --force option to bypass the >>>>> user_input >>>>> would be helpful (at least for test automation): >>>>> >>>>> + if not ipautil.user_input("Continue to clean?", False): >>>>> + sys.exit("Aborted") >>>> >>>> Yup, added. >>>> >>>> rob >>> >>> Slightly revised patch. I still had a window open with one unsaved change. >>> >>> rob >>> >> >> Apparently there were two unsaved changes, one of which was lost. This adds in >> the 'entry already exists' fix. >> >> rob >> > > Just one last thing (otherwise the patch is OK) - I don't think this is what we > want :-) > > # ipa-replica-manage clean-ruv 8 > Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 > > Cleaning the wrong replica ID will cause that server to no > longer replicate so it may miss updates while the process > is running. It would need to be re-initialized to maintain > consistency. Be very careful. > Continue to clean? [no]: y <<<<<< > Aborted > > > Nor this exception, (your are checking for wrong exception): > > # ipa-replica-manage clean-ruv 8 > Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 > > Cleaning the wrong replica ID will cause that server to no > longer replicate so it may miss updates while the process > is running. It would need to be re-initialized to maintain > consistency. Be very careful. > Continue to clean? [no]: > unexpected error: This entry already exists > > This is the exception: > > Traceback (most recent call last): > File "/sbin/ipa-replica-manage", line 651, in > main() > File "/sbin/ipa-replica-manage", line 648, in main > clean_ruv(realm, args[1], options) > File "/sbin/ipa-replica-manage", line 373, in clean_ruv > thisrepl.cleanallruv(ruv) > File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", > line 1136, in cleanallruv > self.conn.addEntry(e) > File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 503, in > addEntry > self.__handle_errors(e, arg_desc=arg_desc) > File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 321, in > __handle_errors > raise errors.DuplicateEntry() > ipalib.errors.DuplicateEntry: This entry already exists > > Martin > Fixed that and a couple of other problems. When doing a disconnect we should not also call clean-ruv. I also got tired of seeing crappy error messages so I added a little convert utility. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-rcrit-1031-8-cleanruv.patch Type: text/x-diff Size: 21575 bytes Desc: not available URL: From rcritten at redhat.com Thu Sep 6 21:22:28 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 06 Sep 2012 17:22:28 -0400 Subject: [Freeipa-devel] [PATCH] 1050 prevent replica orphans In-Reply-To: <5048BFAD.1070807@redhat.com> References: <5040F6E2.7060109@redhat.com> <5040F722.4060208@redhat.com> <5048BFAD.1070807@redhat.com> Message-ID: <50491414.2070504@redhat.com> Martin Kosek wrote: > On 08/31/2012 07:40 PM, Rob Crittenden wrote: >> Rob Crittenden wrote: >>> It was possible use ipa-replica-manage connect/disconnect/del to end up >>> orphaning or or more IPA masters. This is an attempt to catch and >>> prevent that case. >>> >>> I tested with this topology, trying to delete B. >>> >>> A <-> B <-> C >>> >>> I got here by creating B and C from A, connecting B to C then deleting >>> the link from A to B, so it went from A -> B and A -> C to the above. >>> >>> What I do is look up the servers that the delete candidate host has >>> connections to and see if we're the last link. >>> >>> I added an escape clause if there are only two masters. >>> >>> rob >> >> Oh, this relies on my cleanruv patch 1031. >> >> rob >> > > 1) When I run ipa-replica-manage del --force to an already uninstalled host, > the new code will prevent me the deletation because it cannot connect to it. It > also crashes with UnboundLocalError: > > # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force > > Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing removal > Traceback (most recent call last): > File "/sbin/ipa-replica-manage", line 708, in > main() > File "/sbin/ipa-replica-manage", line 677, in main > del_master(realm, args[1], options) > File "/sbin/ipa-replica-manage", line 476, in del_master > sys.exit("Failed read master data from '%s': %s" % (delrepl.hostname, str(e))) > UnboundLocalError: local variable 'delrepl' referenced before assignment Fixed. > > > I also hit this error when removing a winsync replica. Fixed. > > > 2) As I wrote before, I think having --force option override the user inquiries > would benefit test automation: > > + if not ipautil.user_input("Continue to delete?", False): > + sys.exit("Aborted") Fixed. > > > 3) I don't think this code won't cover this topology: > > A - B - C - D - E > > It would allow you deleting a replica C even though it would separate A-B and > D-E. Though we may not want to cover this situation now, what you got is > definitely helping. I think you may be right. I only tested with 4 servers. With this B and D would both still have 2 agreements so wouldn't be covered by the last link test. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-rcrit-1050-2-replicaorphan.patch Type: text/x-diff Size: 6380 bytes Desc: not available URL: From mkosek at redhat.com Fri Sep 7 08:34:07 2012 From: mkosek at redhat.com (Martin Kosek) Date: Fri, 07 Sep 2012 10:34:07 +0200 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <504912F9.1020907@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> <50479491.1060200@redhat.com> <50488DD0.1080507@redhat.com> <5048BEF8.5000401@redhat.com> <5048C46A.8020207@redhat.com> <5048C754.2030708@redhat.com> <5048C9BE.1050005@redhat.com> <504912F9.1020907@redhat.com> Message-ID: <1347006847.2829.10.camel@priserak> On Thu, 2012-09-06 at 17:17 -0400, Rob Crittenden wrote: > Martin Kosek wrote: > > On 09/06/2012 05:55 PM, Rob Crittenden wrote: > >> Rob Crittenden wrote: > >>> Rob Crittenden wrote: > >>>> Martin Kosek wrote: > >>>>> On 09/05/2012 08:06 PM, Rob Crittenden wrote: > >>>>>> Rob Crittenden wrote: > >>>>>>> Martin Kosek wrote: > >>>>>>>> On 07/05/2012 08:39 PM, Rob Crittenden wrote: > >>>>>>>>> Martin Kosek wrote: > >>>>>>>>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: > >>>>>>>>>>> Deleting a replica can leave a replication vector (RUV) on the > >>>>>>>>>>> other servers. > >>>>>>>>>>> This can confuse things if the replica is re-added, and it also > >>>>>>>>>>> causes the > >>>>>>>>>>> server to calculate changes against a server that may no longer > >>>>>>>>>>> exist. > >>>>>>>>>>> > >>>>>>>>>>> 389-ds-base provides a new task that self-propogates itself to all > >>>>>>>>>>> available > >>>>>>>>>>> replicas to clean this RUV data. > >>>>>>>>>>> > >>>>>>>>>>> This patch will create this task at deletion time to hopefully > >>>>>>>>>>> clean things up. > >>>>>>>>>>> > >>>>>>>>>>> It isn't perfect. If any replica is down or unavailable at the > >>>>>>>>>>> time > >>>>>>>>>>> the > >>>>>>>>>>> cleanruv task fires, and then comes back up, the old RUV data > >>>>>>>>>>> may be > >>>>>>>>>>> re-propogated around. > >>>>>>>>>>> > >>>>>>>>>>> To make things easier in this case I've added two new commands to > >>>>>>>>>>> ipa-replica-manage. The first lists the replication ids of all the > >>>>>>>>>>> servers we > >>>>>>>>>>> have a RUV for. Using this you can call clean_ruv with the > >>>>>>>>>>> replication id of a > >>>>>>>>>>> server that no longer exists to try the cleanallruv step again. > >>>>>>>>>>> > >>>>>>>>>>> This is quite dangerous though. If you run cleanruv against a > >>>>>>>>>>> replica id that > >>>>>>>>>>> does exist it can cause a loss of data. I believe I've put in > >>>>>>>>>>> enough scary > >>>>>>>>>>> warnings about this. > >>>>>>>>>>> > >>>>>>>>>>> rob > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Good work there, this should make cleaning RUVs much easier than > >>>>>>>>>> with the > >>>>>>>>>> previous version. > >>>>>>>>>> > >>>>>>>>>> This is what I found during review: > >>>>>>>>>> > >>>>>>>>>> 1) list_ruv and clean_ruv command help in man is quite lost. I > >>>>>>>>>> think > >>>>>>>>>> it would > >>>>>>>>>> help if we for example have all info for commands indented. This > >>>>>>>>>> way > >>>>>>>>>> user could > >>>>>>>>>> simply over-look the new commands in the man page. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> 2) I would rename new commands to clean-ruv and list-ruv to make > >>>>>>>>>> them > >>>>>>>>>> consistent with the rest of the commands (re-initialize, > >>>>>>>>>> force-sync). > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> 3) It would be nice to be able to run clean_ruv command in an > >>>>>>>>>> unattended way > >>>>>>>>>> (for better testing), i.e. respect --force option as we already > >>>>>>>>>> do for > >>>>>>>>>> ipa-replica-manage del. This fix would aid test automation in the > >>>>>>>>>> future. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> 4) (minor) The new question (and the del too) does not react too > >>>>>>>>>> well for > >>>>>>>>>> CTRL+D: > >>>>>>>>>> > >>>>>>>>>> # ipa-replica-manage clean_ruv 3 --force > >>>>>>>>>> Clean the Replication Update Vector for > >>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 > >>>>>>>>>> > >>>>>>>>>> Cleaning the wrong replica ID will cause that server to no > >>>>>>>>>> longer replicate so it may miss updates while the process > >>>>>>>>>> is running. It would need to be re-initialized to maintain > >>>>>>>>>> consistency. Be very careful. > >>>>>>>>>> Continue to clean? [no]: unexpected error: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> 5) Help for clean_ruv command without a required parameter is quite > >>>>>>>>>> confusing > >>>>>>>>>> as it reports that command is wrong and not the parameter: > >>>>>>>>>> > >>>>>>>>>> # ipa-replica-manage clean_ruv > >>>>>>>>>> Usage: ipa-replica-manage [options] > >>>>>>>>>> > >>>>>>>>>> ipa-replica-manage: error: must provide a command [clean_ruv | > >>>>>>>>>> force-sync | > >>>>>>>>>> disconnect | connect | del | re-initialize | list | list_ruv] > >>>>>>>>>> > >>>>>>>>>> It seems you just forgot to specify the error message in the > >>>>>>>>>> command > >>>>>>>>>> definition > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> 6) When the remote replica is down, the clean_ruv command fails > >>>>>>>>>> with an > >>>>>>>>>> unexpected error: > >>>>>>>>>> > >>>>>>>>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 > >>>>>>>>>> Clean the Replication Update Vector for > >>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 > >>>>>>>>>> > >>>>>>>>>> Cleaning the wrong replica ID will cause that server to no > >>>>>>>>>> longer replicate so it may miss updates while the process > >>>>>>>>>> is running. It would need to be re-initialized to maintain > >>>>>>>>>> consistency. Be very careful. > >>>>>>>>>> Continue to clean? [no]: y > >>>>>>>>>> unexpected error: {'desc': 'Operations error'} > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: > >>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - > >>>>>>>>>> cleanAllRUV_task: failed > >>>>>>>>>> to connect to repl agreement connection > >>>>>>>>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, > >>>>>>>>>> > >>>>>>>>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> tree,cn=config), error 105 > >>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - > >>>>>>>>>> cleanAllRUV_task: replica > >>>>>>>>>> (cn=meTovm-055.idm.lab. > >>>>>>>>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> tree, cn=config) has not been cleaned. You will need to rerun > >>>>>>>>>> the > >>>>>>>>>> CLEANALLRUV task on this replica. > >>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - > >>>>>>>>>> cleanAllRUV_task: Task > >>>>>>>>>> failed (1) > >>>>>>>>>> > >>>>>>>>>> In this case I think we should inform user that the command failed, > >>>>>>>>>> possibly > >>>>>>>>>> because of disconnected replicas and that they could enable the > >>>>>>>>>> replicas and > >>>>>>>>>> try again. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> 7) (minor) "pass" is now redundant in replication.py: > >>>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: > >>>>>>>>>> + # We can't make the server we're removing read-only > >>>>>>>>>> but > >>>>>>>>>> + # this isn't a show-stopper > >>>>>>>>>> + root_logger.debug("No permission to switch replica to > >>>>>>>>>> read-only, > >>>>>>>>>> continuing anyway") > >>>>>>>>>> + pass > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> I think this addresses everything. > >>>>>>>>> > >>>>>>>>> rob > >>>>>>>> > >>>>>>>> Thanks, almost there! I just found one more issue which needs to be > >>>>>>>> fixed > >>>>>>>> before we push: > >>>>>>>> > >>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force > >>>>>>>> Directory Manager password: > >>>>>>>> > >>>>>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing > >>>>>>>> removal > >>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': > >>>>>>>> "Can't > >>>>>>>> contact LDAP server"} > >>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' > >>>>>>>> > >>>>>>>> There were issues removing a connection: %d format: a number is > >>>>>>>> required, not str > >>>>>>>> > >>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': > >>>>>>>> "Can't > >>>>>>>> contact LDAP server"} > >>>>>>>> > >>>>>>>> This is a traceback I retrieved: > >>>>>>>> Traceback (most recent call last): > >>>>>>>> File "/sbin/ipa-replica-manage", line 425, in del_master > >>>>>>>> del_link(realm, r, hostname, options.dirman_passwd, force=True) > >>>>>>>> File "/sbin/ipa-replica-manage", line 271, in del_link > >>>>>>>> repl1.cleanallruv(replica_id) > >>>>>>>> File > >>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", > >>>>>>>> line 1094, in cleanallruv > >>>>>>>> root_logger.debug("Creating CLEANALLRUV task for replica id > >>>>>>>> %d" % > >>>>>>>> replicaId) > >>>>>>>> > >>>>>>>> > >>>>>>>> The problem here is that you don't convert replica_id to int in this > >>>>>>>> part: > >>>>>>>> + replica_id = None > >>>>>>>> + if repl2: > >>>>>>>> + replica_id = repl2._get_replica_id(repl2.conn, None) > >>>>>>>> + else: > >>>>>>>> + servers = get_ruv(realm, replica1, dirman_passwd) > >>>>>>>> + for (netloc, rid) in servers: > >>>>>>>> + if netloc.startswith(replica2): > >>>>>>>> + replica_id = rid > >>>>>>>> + break > >>>>>>>> > >>>>>>>> Martin > >>>>>>>> > >>>>>>> > >>>>>>> Updated patch using new mechanism in 389-ds-base. This should more > >>>>>>> thoroughly clean out RUV data when a replica is being deleted, and > >>>>>>> provide for a way to delete RUV data afterwards too if necessary. > >>>>>>> > >>>>>>> rob > >>>>>> > >>>>>> Rebased patch > >>>>>> > >>>>>> rob > >>>>>> > >>>>> > >>>>> 0) As I wrote in a review for your patch 1041, changelog entry slipped > >>>>> elsewhere. > >>>>> > >>>>> 1) The following KeyboardInterrupt except class looks suspicious. I > >>>>> know why > >>>>> you have it there, but since it is generally a bad thing to do, some > >>>>> comment > >>>>> why it is needed would be useful. > >>>>> > >>>>> @@ -256,6 +263,17 @@ def del_link(realm, replica1, replica2, > >>>>> dirman_passwd, > >>>>> force=False): > >>>>> repl1.delete_agreement(replica2) > >>>>> repl1.delete_referral(replica2) > >>>>> > >>>>> + if type1 == replication.IPA_REPLICA: > >>>>> + if repl2: > >>>>> + ruv = repl2._get_replica_id(repl2.conn, None) > >>>>> + else: > >>>>> + ruv = get_ruv_by_host(realm, replica1, replica2, > >>>>> dirman_passwd) > >>>>> + > >>>>> + try: > >>>>> + repl1.cleanallruv(ruv) > >>>>> + except KeyboardInterrupt: > >>>>> + pass > >>>>> + > >>>>> > >>>>> Maybe you just wanted to do some cleanup and then "raise" again? > >>>> > >>>> No, it is there because it is safe to break out of it. The task will > >>>> continue to run. I added some verbiage. > >>>> > >>>>> > >>>>> 2) This is related to 1), but when some replica is down, > >>>>> "ipa-replica-manage > >>>>> del" may wait indefinitely when some remote replica is down, right? > >>>>> > >>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com > >>>>> Deleting a master is irreversible. > >>>>> To reconnect to the remote master you will need to prepare a new > >>>>> replica file > >>>>> and re-install. > >>>>> Continue to delete? [no]: y > >>>>> ipa: INFO: Setting agreement > >>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping > >>>>> > >>>>> > >>>>> > >>>>> tree,cn=config schedule to 2358-2359 0 to force synch > >>>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement > >>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping > >>>>> > >>>>> > >>>>> > >>>>> tree,cn=config > >>>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica > >>>>> acquired > >>>>> successfully: Incremental update succeeded: start: 0: end: 0 > >>>>> Background task created to clean replication data > >>>>> > >>>>> ... after about a minute I hit CTRL+C > >>>>> > >>>>> ^CDeleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to > >>>>> 'vm-055.idm.lab.bos.redhat.com' > >>>>> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: NS record > >>>>> does not > >>>>> contain 'vm-055.idm.lab.bos.redhat.com.' > >>>>> You may need to manually remove them from the tree > >>>>> > >>>>> I think it would be better to inform user that some remote replica is > >>>>> down or > >>>>> at least that we are waiting for the task to complete. Something like > >>>>> that: > >>>>> > >>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com > >>>>> ... > >>>>> Background task created to clean replication data > >>>>> Replication data clean up may take very long time if some replica is > >>>>> unreachable > >>>>> Hit CTRL+C to interrupt the wait > >>>>> ^C Clean up wait interrupted > >>>>> .... > >>>>> [continue with del] > >>>> > >>>> Yup, did this in #1. > >>>> > >>>>> > >>>>> 3) (minor) When there is a cleanruv task running and you run > >>>>> "ipa-replica-manage del", there is a unexpected error message with > >>>>> duplicate > >>>>> task object in LDAP: > >>>>> > >>>>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com --force > >>>>> Unable to connect to replica vm-072.idm.lab.bos.redhat.com, forcing > >>>>> removal > >>>>> FAIL > >>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't > >>>>> contact LDAP server"} > >>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' > >>>>> > >>>>> There were issues removing a connection: This entry already exists > >>>>> <<<<<<<<< > >>>>> > >>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't > >>>>> contact LDAP server"} > >>>>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record > >>>>> does not > >>>>> contain 'vm-072.idm.lab.bos.redhat.com.' > >>>>> You may need to manually remove them from the tree > >>>>> > >>>>> > >>>>> I think it should be enough to just catch for "entry already exists" in > >>>>> cleanallruv function, and in such case print a relevant error message > >>>>> bail out. > >>>>> Thus, self.conn.checkTask(dn, dowait=True) would not be called too. > >>>> > >>>> Good catch, fixed. > >>>> > >>>>> > >>>>> > >>>>> 4) (minor): In make_readonly function, there is a redundant "pass" > >>>>> statement: > >>>>> > >>>>> + def make_readonly(self): > >>>>> + """ > >>>>> + Make the current replication agreement read-only. > >>>>> + """ > >>>>> + dn = DN(('cn', 'userRoot'), ('cn', 'ldbm database'), > >>>>> + ('cn', 'plugins'), ('cn', 'config')) > >>>>> + > >>>>> + mod = [(ldap.MOD_REPLACE, 'nsslapd-readonly', 'on')] > >>>>> + try: > >>>>> + self.conn.modify_s(dn, mod) > >>>>> + except ldap.INSUFFICIENT_ACCESS: > >>>>> + # We can't make the server we're removing read-only but > >>>>> + # this isn't a show-stopper > >>>>> + root_logger.debug("No permission to switch replica to > >>>>> read-only, > >>>>> continuing anyway") > >>>>> + pass <<<<<<<<<<<<<<< > >>>> > >>>> Yeah, this is one of my common mistakes. I put in a pass initially, then > >>>> add logging in front of it and forget to delete the pass. Its gone now. > >>>> > >>>>> > >>>>> > >>>>> 5) In clean_ruv, I think allowing a --force option to bypass the > >>>>> user_input > >>>>> would be helpful (at least for test automation): > >>>>> > >>>>> + if not ipautil.user_input("Continue to clean?", False): > >>>>> + sys.exit("Aborted") > >>>> > >>>> Yup, added. > >>>> > >>>> rob > >>> > >>> Slightly revised patch. I still had a window open with one unsaved change. > >>> > >>> rob > >>> > >> > >> Apparently there were two unsaved changes, one of which was lost. This adds in > >> the 'entry already exists' fix. > >> > >> rob > >> > > > > Just one last thing (otherwise the patch is OK) - I don't think this is what we > > want :-) > > > > # ipa-replica-manage clean-ruv 8 > > Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 > > > > Cleaning the wrong replica ID will cause that server to no > > longer replicate so it may miss updates while the process > > is running. It would need to be re-initialized to maintain > > consistency. Be very careful. > > Continue to clean? [no]: y <<<<<< > > Aborted > > > > > > Nor this exception, (your are checking for wrong exception): > > > > # ipa-replica-manage clean-ruv 8 > > Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 > > > > Cleaning the wrong replica ID will cause that server to no > > longer replicate so it may miss updates while the process > > is running. It would need to be re-initialized to maintain > > consistency. Be very careful. > > Continue to clean? [no]: > > unexpected error: This entry already exists > > > > This is the exception: > > > > Traceback (most recent call last): > > File "/sbin/ipa-replica-manage", line 651, in > > main() > > File "/sbin/ipa-replica-manage", line 648, in main > > clean_ruv(realm, args[1], options) > > File "/sbin/ipa-replica-manage", line 373, in clean_ruv > > thisrepl.cleanallruv(ruv) > > File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", > > line 1136, in cleanallruv > > self.conn.addEntry(e) > > File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 503, in > > addEntry > > self.__handle_errors(e, arg_desc=arg_desc) > > File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 321, in > > __handle_errors > > raise errors.DuplicateEntry() > > ipalib.errors.DuplicateEntry: This entry already exists > > > > Martin > > > > Fixed that and a couple of other problems. When doing a disconnect we > should not also call clean-ruv. > > I also got tired of seeing crappy error messages so I added a little > convert utility. > > rob ACK. Do you want to want with the pushing until the winsync issue in CLEANALLRUV is fixed? This is the ticket I created: https://fedorahosted.org/389/ticket/450 Martin From mkosek at redhat.com Fri Sep 7 09:05:04 2012 From: mkosek at redhat.com (Martin Kosek) Date: Fri, 07 Sep 2012 11:05:04 +0200 Subject: [Freeipa-devel] [PATCH] 1050 prevent replica orphans In-Reply-To: <50491414.2070504@redhat.com> References: <5040F6E2.7060109@redhat.com> <5040F722.4060208@redhat.com> <5048BFAD.1070807@redhat.com> <50491414.2070504@redhat.com> Message-ID: <1347008704.2829.13.camel@priserak> On Thu, 2012-09-06 at 17:22 -0400, Rob Crittenden wrote: > Martin Kosek wrote: > > On 08/31/2012 07:40 PM, Rob Crittenden wrote: > >> Rob Crittenden wrote: > >>> It was possible use ipa-replica-manage connect/disconnect/del to end up > >>> orphaning or or more IPA masters. This is an attempt to catch and > >>> prevent that case. > >>> > >>> I tested with this topology, trying to delete B. > >>> > >>> A <-> B <-> C > >>> > >>> I got here by creating B and C from A, connecting B to C then deleting > >>> the link from A to B, so it went from A -> B and A -> C to the above. > >>> > >>> What I do is look up the servers that the delete candidate host has > >>> connections to and see if we're the last link. > >>> > >>> I added an escape clause if there are only two masters. > >>> > >>> rob > >> > >> Oh, this relies on my cleanruv patch 1031. > >> > >> rob > >> > > > > 1) When I run ipa-replica-manage del --force to an already uninstalled host, > > the new code will prevent me the deletation because it cannot connect to it. It > > also crashes with UnboundLocalError: > > > > # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force > > > > Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing removal > > Traceback (most recent call last): > > File "/sbin/ipa-replica-manage", line 708, in > > main() > > File "/sbin/ipa-replica-manage", line 677, in main > > del_master(realm, args[1], options) > > File "/sbin/ipa-replica-manage", line 476, in del_master > > sys.exit("Failed read master data from '%s': %s" % (delrepl.hostname, str(e))) > > UnboundLocalError: local variable 'delrepl' referenced before assignment > > Fixed. > > > > > > > I also hit this error when removing a winsync replica. > > Fixed. > > > > > > > 2) As I wrote before, I think having --force option override the user inquiries > > would benefit test automation: > > > > + if not ipautil.user_input("Continue to delete?", False): > > + sys.exit("Aborted") > > Fixed. > > > > > > > 3) I don't think this code won't cover this topology: > > > > A - B - C - D - E > > > > It would allow you deleting a replica C even though it would separate A-B and > > D-E. Though we may not want to cover this situation now, what you got is > > definitely helping. > > I think you may be right. I only tested with 4 servers. With this B and > D would both still have 2 agreements so wouldn't be covered by the last > link test. Everything looks good now, so ACK. We just need to push it along with CLEANALLRUV patch. Martin From mkosek at redhat.com Fri Sep 7 11:00:38 2012 From: mkosek at redhat.com (Martin Kosek) Date: Fri, 07 Sep 2012 13:00:38 +0200 Subject: [Freeipa-devel] [PATCH] 1052 add version to prepared replica files In-Reply-To: <5040F91B.6000606@redhat.com> References: <5040F91B.6000606@redhat.com> Message-ID: <1347015638.2047.1.camel@priserak> On Fri, 2012-08-31 at 13:49 -0400, Rob Crittenden wrote: > When installing a replica in an upgrade situation we want to be sure we > install the same version or higher. This will have to bake a bit until > the next full version of IPA but the idea is to prevent installing a > newer replica file on an older server. > > To test this you need to rip apart a prepared file and tweak the version > forward or backward. > > To do this, do something like: > > # gpg -d replica-info-pitbull.example.com.gpg | tar xf - > # edit realm_info/realm_info > # tar cf replica-info-pitbull.example.com realm_info > # gpg --batch --homedir `pwd`/.gnupg --passphrase-fd 0 --yes --no-tty -o > replica-info-pitbull.example.com.gpg -c replica-info-pitbull.example.com > > > rob Works fine. ACK. Pushed to master, ipa-3-0. Martin From pspacek at redhat.com Fri Sep 7 11:05:37 2012 From: pspacek at redhat.com (Petr Spacek) Date: Fri, 07 Sep 2012 13:05:37 +0200 Subject: [Freeipa-devel] [PATCH 0055] Fix race condition in addrdataset() during SOA serial update Message-ID: <5049D501.20207@redhat.com> Hello, Fix race condition in addrdataset() during SOA serial update. https://fedorahosted.org/bind-dyndb-ldap/ticket/89 -- Petr^2 Spacek -------------- next part -------------- A non-text attachment was scrubbed... Name: bind-dyndb-ldap-pspacek-0055-Fix-race-condition-in-addrdataset-during-SOA-serial-.patch Type: text/x-patch Size: 4150 bytes Desc: not available URL: From mkosek at redhat.com Fri Sep 7 11:40:18 2012 From: mkosek at redhat.com (Martin Kosek) Date: Fri, 07 Sep 2012 13:40:18 +0200 Subject: [Freeipa-devel] [PATCH] 1046 add e-mail by default In-Reply-To: <5048C44F.3030106@redhat.com> References: <5037BFEB.8050602@redhat.com> <50489AE2.7020709@redhat.com> <5048C44F.3030106@redhat.com> Message-ID: <1347018018.2047.4.camel@priserak> On Thu, 2012-09-06 at 11:42 -0400, Rob Crittenden wrote: > Martin Kosek wrote: > > On 08/24/2012 07:54 PM, Rob Crittenden wrote: > >> We weren't automatically creating the mail attribute despite having the default > >> e-mail domain. This patch will add it to all new users. > >> > >> To disable creating this set the default e-mail domain to empty in ipa config. > >> > >> rob > >> > > > > 1) Patch needs a rebase > > > > 2) There are 2 test cases where new default mail attribute was not added: > > > > ====================================================================== > > FAIL: test_user[34]: user_find: Search for "tuser2" with manager "tuser1" > > ---------------------------------------------------------------------- > > ... > > extra keys = ['mail'] > > ... > > > > ====================================================================== > > FAIL: test_user[75]: user_add: Create 2nd admin user "admin2" > > ---------------------------------------------------------------------- > > ... > > extra keys = ['mail'] > > ... > > > > 3) Some code could be simplified: > > > > This: > > + if 'ipadefaultemaildomain' in config: > > + defaultdomain = config['ipadefaultemaildomain'][0] > > + else: > > + defaultdomain = None > > > > To this: > > defaultdomain = config.get('ipadefaultemaildomain', [None])[0] > > > > > > This: > > if m.find('@') == -1 ... > > > > To this: > > if '@' not in m ... > > > > IMHO, it is more readable than the find method. > > > > 3) When default e-mail domain is removed from config, users cannot be added any > > more when e-mail is not explicitly specified: > > > > # ipa config-mod --emaildomain= > > Maximum username length: 32 > > Home directory base: /home > > Default shell: /bin/sh > > Default users group: ipausers > > Search time limit: 2 > > Search size limit: 100 > > User search fields: uid,givenname,sn,telephonenumber,ou,title > > Group search fields: cn,description > > Enable migration mode: FALSE > > Certificate Subject base: O=IDM.LAB.BOS.REDHAT.COM > > Password Expiration Notification (days): 4 > > Password plugin features: AllowNThash > > SELinux user map order: > > guest_u:s0$xguest_u:s0$user_u:s0-s0:c0.c1023$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023 > > Default SELinux user: guest_u:s0 > > PAC type: MS-PAC > > > > # ipa user-add --first=Foo --last=Bar fbar > > ipa: ERROR: invalid 'email': invalid e-mail format: fbar > > > > Martin > > > > Rebased, issues addressed. > > rob Yup, its OK now. ACK. I just had to squash an attached one-liner which updates range tests I committed yesterday. Pushed to master, ipa-3-0. Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Add-missing-mail-attribute-to-range-tests.patch Type: text/x-patch Size: 1051 bytes Desc: not available URL: From mkosek at redhat.com Fri Sep 7 11:54:18 2012 From: mkosek at redhat.com (Martin Kosek) Date: Fri, 07 Sep 2012 13:54:18 +0200 Subject: [Freeipa-devel] [PATCH] 305 Update DNS zone allow-query validation test In-Reply-To: <5048904F.6060802@redhat.com> References: <50487215.70600@redhat.com> <50488A8E.4030409@redhat.com> <5048904F.6060802@redhat.com> Message-ID: <1347018858.2047.6.camel@priserak> On Thu, 2012-09-06 at 14:00 +0200, Martin Kosek wrote: > On 09/06/2012 01:35 PM, Petr Vobornik wrote: > > On 09/06/2012 11:51 AM, Martin Kosek wrote: > >> Loopback address, "localhost" and "localnets" ACIs are no longer > >> an issue for bind-dyndb-ldap. Allow them in our validators. > >> > > > > Martin's patch works and looks good - ACK. > > > > > > Attaching patch for Web UI part. > > Web UI validator works fine too, ACK. > > Pushed both patches to master, ipa-3-0. > > Martin I forgot to fix one DNS zone unit test. Fixed (patch attached) and pushed as a one(two)-liner. Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-305-update-dns-zone-allow-query-validation-test.patch Type: text/x-patch Size: 1215 bytes Desc: not available URL: From sbose at redhat.com Fri Sep 7 12:02:01 2012 From: sbose at redhat.com (Sumit Bose) Date: Fri, 7 Sep 2012 14:02:01 +0200 Subject: [Freeipa-devel] [PATCH] Set master_kdc and dns_lookup_kdc to true Message-ID: <20120907120201.GC30746@localhost.localdomain> Hi, those two patches should fix https://fedorahosted.org/freeipa/ticket/2515 . The first makes the needed change for fresh installations. The second adds the changes during ipa-adtrust-install if needed. I prefer to do the changes here instead of during updates, because during updates it is not easy to see that the Kerberos configuration was changes. bye, Sumit -------------- next part -------------- From af51c4e31fe691a05498c29d334b5958c60dface Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Thu, 16 Aug 2012 13:16:55 +0200 Subject: [PATCH 67/68] Set master_kdc and dns_lookup_kdc to true --- contrib/RHEL4/ipa-client-setup | 3 ++- install/share/krb5.conf.template | 3 ++- install/share/krb5.ini.template | 1 + install/tools/ipa-replica-conncheck | 3 ++- ipa-client/ipa-install/ipa-client-install | 1 + 5 Dateien ge?ndert, 8 Zeilen hinzugef?gt(+), 3 Zeilen entfernt(-) diff --git a/contrib/RHEL4/ipa-client-setup b/contrib/RHEL4/ipa-client-setup index 1a8761036e1b7230b1524c45d565126ff73030b4..4d1fead981d0e10232e974527222a2f9a62252b4 100644 --- a/contrib/RHEL4/ipa-client-setup +++ b/contrib/RHEL4/ipa-client-setup @@ -307,7 +307,7 @@ def main(): #[libdefaults] libopts = [{'name':'default_realm', 'type':'option', 'value':ipasrv.getRealmName()}] libopts.append({'name':'dns_lookup_realm', 'type':'option', 'value':'false'}) - libopts.append({'name':'dns_lookup_kdc', 'type':'option', 'value':'false'}) + libopts.append({'name':'dns_lookup_kdc', 'type':'option', 'value':'true'}) libopts.append({'name':'ticket_lifetime', 'type':'option', 'value':'24h'}) libopts.append({'name':'forwardable', 'type':'option', 'value':'yes'}) @@ -316,6 +316,7 @@ def main(): #[realms] kropts =[{'name':'kdc', 'type':'option', 'value':ipasrv.getServerName()+':88'}, + {'name':'master_kdc', 'type':'option', 'value':ipasrv.getServerName()+':88'}, {'name':'admin_server', 'type':'option', 'value':ipasrv.getServerName()+':749'}, {'name':'default_domain', 'type':'option', 'value':ipasrv.getDomainName()}] ropts = [{'name':ipasrv.getRealmName(), 'type':'subsection', 'value':kropts}] diff --git a/install/share/krb5.conf.template b/install/share/krb5.conf.template index eda8ba6fe647d54d5feef1acda41c482b0dbcefa..f8b1a6f09868c55e47f21279b6d061fbd8251171 100644 --- a/install/share/krb5.conf.template +++ b/install/share/krb5.conf.template @@ -6,7 +6,7 @@ [libdefaults] default_realm = $REALM dns_lookup_realm = false - dns_lookup_kdc = false + dns_lookup_kdc = true rdns = false ticket_lifetime = 24h forwardable = yes @@ -14,6 +14,7 @@ [realms] $REALM = { kdc = $FQDN:88 + master_kdc = $FQDN:88 admin_server = $FQDN:749 default_domain = $DOMAIN pkinit_anchors = FILE:/etc/ipa/ca.crt diff --git a/install/share/krb5.ini.template b/install/share/krb5.ini.template index 89f4a370143ac0848b7eeed24085d897242595f1..01cc1369f518f8e903d175d5c41e40040eaa1784 100644 --- a/install/share/krb5.ini.template +++ b/install/share/krb5.ini.template @@ -8,6 +8,7 @@ $REALM = { admin_server = $FQDN kdc = $FQDN + master_kdc = $FQDN default_domain = $REALM } diff --git a/install/tools/ipa-replica-conncheck b/install/tools/ipa-replica-conncheck index 8e4536cf67cafb907a3e330607a81b4bc034015b..169e9dc9f1d28dcc7c36b09f4382b8948d5ae831 100755 --- a/install/tools/ipa-replica-conncheck +++ b/install/tools/ipa-replica-conncheck @@ -177,7 +177,7 @@ def configure_krb5_conf(realm, kdc, filename): #[libdefaults] libdefaults = [{'name':'default_realm', 'type':'option', 'value':realm}] libdefaults.append({'name':'dns_lookup_realm', 'type':'option', 'value':'false'}) - libdefaults.append({'name':'dns_lookup_kdc', 'type':'option', 'value':'false'}) + libdefaults.append({'name':'dns_lookup_kdc', 'type':'option', 'value':'true'}) libdefaults.append({'name':'rdns', 'type':'option', 'value':'false'}) libdefaults.append({'name':'ticket_lifetime', 'type':'option', 'value':'24h'}) libdefaults.append({'name':'forwardable', 'type':'option', 'value':'yes'}) @@ -188,6 +188,7 @@ def configure_krb5_conf(realm, kdc, filename): #the following are necessary only if DNS discovery does not work #[realms] realms_info =[{'name':'kdc', 'type':'option', 'value':ipautil.format_netloc(kdc, 88)}, + {'name':'master_kdc', 'type':'option', 'value':ipautil.format_netloc(kdc, 88)}, {'name':'admin_server', 'type':'option', 'value':ipautil.format_netloc(kdc, 749)}] realms = [{'name':realm, 'type':'subsection', 'value':realms_info}] diff --git a/ipa-client/ipa-install/ipa-client-install b/ipa-client/ipa-install/ipa-client-install index d87fcc2a662b73c8ff269b65437d7d3023509b62..38b632220a1397b73acc042bd343b7638eb96230 100755 --- a/ipa-client/ipa-install/ipa-client-install +++ b/ipa-client/ipa-install/ipa-client-install @@ -671,6 +671,7 @@ def configure_krb5_conf(fstore, cli_basedn, cli_realm, cli_domain, cli_server, c #[realms] for server in cli_server: kropts.append({'name':'kdc', 'type':'option', 'value':ipautil.format_netloc(server, 88)}) + kropts.append({'name':'master_kdc', 'type':'option', 'value':ipautil.format_netloc(server, 88)}) kropts.append({'name':'admin_server', 'type':'option', 'value':ipautil.format_netloc(server, 749)}) kropts.append({'name':'default_domain', 'type':'option', 'value':cli_domain}) kropts.append({'name':'pkinit_anchors', 'type':'option', 'value':'FILE:/etc/ipa/ca.crt'}) -- 1.7.11.4 -------------- next part -------------- From 8328a84bbfeacf95231956c112e970035f367bb9 Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Fri, 7 Sep 2012 12:40:58 +0200 Subject: [PATCH 68/68] Update krb5.conf during ipa-adtrust-install --- ipaserver/install/adtrustinstance.py | 62 ++++++++++++++++++++++++++++++++++++ 1 Datei ge?ndert, 62 Zeilen hinzugef?gt(+) diff --git a/ipaserver/install/adtrustinstance.py b/ipaserver/install/adtrustinstance.py index 078c54dbe6ab520e5e3e7e186d4059b5a0fa252b..a23354c0ddb648d74be7fd8170e38da3a116c18e 100644 --- a/ipaserver/install/adtrustinstance.py +++ b/ipaserver/install/adtrustinstance.py @@ -36,8 +36,11 @@ from ipapython.ipa_log_manager import * from ipapython import services as ipaservices from ipapython.dn import DN +import ipaclient.ipachangeconf + import string import struct +import re ALLOWED_NETBIOS_CHARS = string.ascii_uppercase + string.digits @@ -100,6 +103,7 @@ class ADTRUSTInstance(service.Service): def __init__(self, fstore=None): self.fqdn = None self.ip_address = None + self.realm = None self.domain_name = None self.netbios_name = None self.no_msdcs = None @@ -410,6 +414,63 @@ class ADTRUSTInstance(service.Service): except: self.print_msg(SELINUX_WARNING % dict(var=','.join(sebools))) + def __mod_krb5_conf(self): + """ + Set dns_lookup_kdc to true and master_kdc in /etc/krb5.conf + """ + + if not self.fqdn or not self.realm: + self.print_msg("Cannot modify /etc/krb5.conf") + + krbconf = ipaclient.ipachangeconf.IPAChangeConf("IPA Installer") + krbconf.setOptionAssignment(" = ") + krbconf.setSectionNameDelimiters(("[", "]")) + krbconf.setSubSectionDelimiters(("{", "}")) + krbconf.setIndent(("", " ", " ")) + + libopts = [{'name':'dns_lookup_kdc', 'type':'option', 'action':'set', + 'value':'true'}] + + master_kdc = self.fqdn + ":88" + kropts = [{'name':'master_kdc', 'type':'option', 'action':'set', + 'value':master_kdc}] + + ropts = [{'name':self.realm, 'type':'subsection', 'action':'set', + 'value':kropts}] + + opts = [{'name':'libdefaults', 'type':'section', 'action':'set', + 'value':libopts}, + {'name':'realms', 'type':'section', 'action':'set', + 'value':ropts}] + + krbconf.changeConf("/etc/krb5.conf", opts) + + def __update_krb5_conf(self): + """ + Update /etc/krb5.conf if needed + """ + + try: + krb5conf = open("/etc/krb5.conf", 'r') + except IOError, e: + self.print_msg("Cannot open /etc/krb5.conf (%s)\n" % str(e)) + return + + has_dns_lookup_kdc_true = False + for line in krb5conf: + if re.match("^\s*dns_lookup_kdc\s*=\s*[Tt][Rr][Uu][Ee]\s*$", line): + has_dns_lookup_kdc_true = True + break + krb5conf.close() + + if not has_dns_lookup_kdc_true: + self.__mod_krb5_conf() + else: + self.print_msg("'dns_lookup_kdc' already set to 'true', " + "nothing to do.") + + + def __start(self): try: self.start() @@ -541,6 +602,7 @@ class ADTRUSTInstance(service.Service): self.step("adding cifs Kerberos principal", self.__setup_principal) self.step("adding admin(group) SIDs", self.__add_admin_sids) self.step("adding RID bases", self.__add_rid_bases) + self.step("updating Kerberos config", self.__update_krb5_conf) self.step("activating CLDAP plugin", self.__add_cldap_module) self.step("activating sidgen plugin and task", self.__add_sidgen_module) self.step("activating extdom plugin", self.__add_extdom_module) -- 1.7.11.4 From pvoborni at redhat.com Fri Sep 7 12:37:11 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Fri, 07 Sep 2012 14:37:11 +0200 Subject: [Freeipa-devel] [PATCH] 211 Prevent opening of multiple dirty dialogs on navigation Message-ID: <5049EA77.9010308@redhat.com> Facets which performs AJAX call after update refresh (clear dirty state) after calling callback of dirty dialog. It might lead to multiple openings of dirty dialog. Assuming that calling dirty dialog's callback can be evaluated as "dirty state is gone", we can call reset in the callback to prevent the issue. There will be an incorrect state in the facet for a moment. It will be fixed soon on execute of callback of the refresh AJAX call. It is not an issue because it will happen in background. User will be looking on different facet. https://fedorahosted.org/freeipa/ticket/2667 -- Petr Vobornik -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pvoborni-0211-Prevent-opening-of-multiple-dirty-dialogs-on-navigat.patch Type: text/x-patch Size: 1598 bytes Desc: not available URL: From mkosek at redhat.com Fri Sep 7 12:53:47 2012 From: mkosek at redhat.com (Martin Kosek) Date: Fri, 07 Sep 2012 14:53:47 +0200 Subject: [Freeipa-devel] [PATCH] 306 Cast DNS SOA serial maximum boundary to long Message-ID: <1347022427.2047.8.camel@priserak> This will fix i386 builds where the SOA serial value written in API.txt was already of a long type while on x86_64 it was still of an int type. ------ I verified that IPA now builds both on i386 and x86_64 platforms. Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-306-cast-dns-soa-serial-maximum-boundary-to-long.patch Type: text/x-patch Size: 4945 bytes Desc: not available URL: From rcritten at redhat.com Fri Sep 7 13:38:46 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 07 Sep 2012 09:38:46 -0400 Subject: [Freeipa-devel] [PATCH] 306 Cast DNS SOA serial maximum boundary to long In-Reply-To: <1347022427.2047.8.camel@priserak> References: <1347022427.2047.8.camel@priserak> Message-ID: <5049F8E6.5070404@redhat.com> Martin Kosek wrote: > This will fix i386 builds where the SOA serial value written > in API.txt was already of a long type while on x86_64 it was still > of an int type. > > ------ > > I verified that IPA now builds both on i386 and x86_64 platforms. > > Martin ACK From mkosek at redhat.com Fri Sep 7 13:41:00 2012 From: mkosek at redhat.com (Martin Kosek) Date: Fri, 07 Sep 2012 15:41:00 +0200 Subject: [Freeipa-devel] [PATCH] 306 Cast DNS SOA serial maximum boundary to long In-Reply-To: <5049F8E6.5070404@redhat.com> References: <1347022427.2047.8.camel@priserak> <5049F8E6.5070404@redhat.com> Message-ID: <1347025260.2047.9.camel@priserak> On Fri, 2012-09-07 at 09:38 -0400, Rob Crittenden wrote: > Martin Kosek wrote: > > This will fix i386 builds where the SOA serial value written > > in API.txt was already of a long type while on x86_64 it was still > > of an int type. > > > > ------ > > > > I verified that IPA now builds both on i386 and x86_64 platforms. > > > > Martin > > ACK > Pushed to master, ipa-3-0. Martin From mkosek at redhat.com Fri Sep 7 15:03:14 2012 From: mkosek at redhat.com (Martin Kosek) Date: Fri, 07 Sep 2012 17:03:14 +0200 Subject: [Freeipa-devel] [PATCHES] Various fixes for trust and range CLI In-Reply-To: <5048B83F.7080309@redhat.com> References: <20120906113947.GD31872@localhost.localdomain> <5048B83F.7080309@redhat.com> Message-ID: <1347030194.2047.12.camel@priserak> On Thu, 2012-09-06 at 16:50 +0200, Petr Vobornik wrote: > On 09/06/2012 01:39 PM, Sumit Bose wrote: > > Hi, > > > > this series of patches touches couple of tickets related to the trust > > and (id)range CLI. I post them together because some of them depend on > > each other. I already rebased them on Martin's "Add range safety check > > for range_mod and range_del" patch which I'm currently reviewing. > > > > bye, > > Sumit > > > > > > ACK for the UI changes in patch 60. > > There is a minor issue in ipa_init_objects.json and > ipa_init_command.json files: Labels aren't changed so there are still > 'Ranges' instead of 'ID ranges'. I don't think it matters because it > doesn't affect actual functionality and I will definitely regenerate > those files sometime in a future. They are only for developer and > related purposes. Ok, I will leave the UI patch as is, it is ok to have the label regenerated later. ACK for the rest of the patches. Pushed them all to master, ipa-3-0. Martin From jcholast at redhat.com Fri Sep 7 15:24:33 2012 From: jcholast at redhat.com (Jan Cholasta) Date: Fri, 07 Sep 2012 17:24:33 +0200 Subject: [Freeipa-devel] [PATCH] 83 Use OpenSSH-style public keys as the preferred format of SSH public keys In-Reply-To: <5048C5A0.5090204@redhat.com> References: <50463229.5030305@redhat.com> <5047BCBE.1090304@redhat.com> <5048C5A0.5090204@redhat.com> Message-ID: <504A11B1.7050003@redhat.com> Dne 6.9.2012 17:47, Jan Cholasta napsal(a): > Dne 5.9.2012 22:57, Rob Crittenden napsal(a): >> Jan Cholasta wrote: >>> Hi, >>> >>> this patch changes the format of the sshpubkey parameter to the format >>> used by OpenSSH (see sshd(8)). >>> >>> Public keys in the old format (raw RFC 4253 blob) are automatically >>> converted to OpenSSH-style public keys. OpenSSH-style public keys are >>> now stored in LDAP. >>> >>> Changed sshpubkeyfp to be an output parameter, as that is what it >>> actually is. >>> >>> Allow parameter normalizers to be used on values of any type, not just >>> unicode, so that public key blobs (which are str) can be normalized to >>> OpenSSH-style public keys. >>> >>> Note that you need a SSSD build including >>> >>> >>> >>> (SSSD 1.9.0beta7-to-be) in order to make OpenSSH integration actually >>> work with OpenSSH-style public keys. >>> >>> >>> >>> >>> Honza >> >> NACK. >> >> I think a bunch of tests are needed for this. >> >> Because you abstracted out the pubkey class it should be straightforward >> to add a bunch of class-based unit tests on it. >> >> There are also no user or host-based tests, either for adding or >> managing keys. > > Tests added. > >> >> I tested backwards compatibility with 2.2 and the initial tests are >> mixed. >> >> I installed 2.2 and created a 3.0 clone from it, including your patch. > > Do people actually do that in real deployments? > >> >> I added a user in 3.0 with a key and it added ok, but on the 2.2 side it >> returns the entire base64 encoded blob of key type, key and comment, >> which I presume is unusable. At least things don't blow up. > > The format of ipasshpubkey in LDAP has changed, so there's not much I > can do about this. > >> >> The reverse works fine. An old-style key added to 2.2 appears to work >> fine in 3.0, we just lack a comment. >> >> On the 2.2 server: >> >> $ ipa user-show tuser1 --all | grep -i ssh >> Base-64 encoded SSH public key: >> c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDNUQyRTI2dHU5YXM2cHhlUVlSdUgzelYyUDUzMjFpR1U5aC9XNElpd0tGSGlOc2p5cXFyemhCUFB3am83dGlYRDlHbUo1M25KS21OTGd0K01XUnFTZEx2R0V3NjM3SkVTWEpGL0VWeUxvZEFWRGltdXFRVkNLWjBRcm1kYjErRUg1VGRrd3ByOExyd0g1a0RzMEVpcGc2c0xoRUZ5NzMvaXNjRkJqcmk0NGxSU1BZNXFHTWFLOVE0cjY1WFEyaytlZ1RDQnBNZnc0b0J6Mzh0ZHVEVVE2bW9XNFhQSnhZeWJ3MGFDMnRUK2RBOU42WndFSFZXREUzdzg0bHRHa0JRZFRaKzViRnBFdlladm9PbkZXdDlNZFIzYVd6UklnY1o5VDlySDFFT2Z3eE5zWVRCLzRjTmg3dS9adGxnMVV0Z1VteWN3TkpMTUYrMTNzNTl2OFFpSFogcmNyaXRAZWRzZWwuZ3JleW9hay5jb20= >> >> >> $ python >> Python 2.7.3 (default, Jul 24 2012, 10:05:38) >> [GCC 4.7.0 20120507 (Red Hat 4.7.0-5)] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> import base64 >> >>> s = >> 'c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDNUQyRTI2dHU5YXM2cHhlUVlSdUgzelYyUDUzMjFpR1U5aC9XNElpd0tGSGlOc2p5cXFyemhCUFB3am83dGlYRDlHbUo1M25KS21OTGd0K01XUnFTZEx2R0V3NjM3SkVTWEpGL0VWeUxvZEFWRGltdXFRVkNLWjBRcm1kYjErRUg1VGRrd3ByOExyd0g1a0RzMEVpcGc2c0xoRUZ5NzMvaXNjRkJqcmk0NGxSU1BZNXFHTWFLOVE0cjY1WFEyaytlZ1RDQnBNZnc0b0J6Mzh0ZHVEVVE2bW9XNFhQSnhZeWJ3MGFDMnRUK2RBOU42WndFSFZXREUzdzg0bHRHa0JRZFRaKzViRnBFdlladm9PbkZXdDlNZFIzYVd6UklnY1o5VDlySDFFT2Z3eE5zWVRCLzRjTmg3dS9adGxnMVV0Z1VteWN3TkpMTUYrMTNzNTl2OFFpSFogcmNyaXRAZWRzZWwuZ3JleW9hay5jb20=' >> >> >> >>> base64.b64decode(s) >> 'ssh-rsa >> AAAAB3NzaC1yc2EAAAADAQABAAABAQC5D2E26tu9as6pxeQYRuH3zV2P5321iGU9h/W4IiwKFHiNsjyqqrzhBPPwjo7tiXD9GmJ53nJKmNLgt+MWRqSdLvGEw637JESXJF/EVyLodAVDimuqQVCKZ0Qrmdb1+EH5Tdkwpr8LrwH5kDs0Eipg6sLhEFy73/iscFBjri44lRSPY5qGMaK9Q4r65XQ2k+egTCBpMfw4oBz38tduDUQ6moW4XPJxYybw0aC2tT+dA9N6ZwEHVWDE3w84ltGkBQdTZ+5bFpEvYZvoOnFWt9MdR3aWzRIgcZ9T9rH1EOfwxNsYTB/4cNh7u/Ztlg1UtgUmycwNJLMF+13s59v8QiHZ >> >> rcrit at edsel.greyoak.com' >> >> Now show an old style key: >> >> $ ipa user-show tuser2 --all | grep -i ssh >> Base-64 encoded SSH public key: >> AAAAB3NzaC1yc2EAAAADAQABAAABAQCbRLyizFGyfucNRnHpWdUG8dBD7W2PfvTQ42k+LmAdUFudTytO89oTRXcVEYMDL42OyRth12JRMUjYTEmFwo9a9Mb7cP8+bo7N2lV4iCB0CUybcZARF0MV6NeYhhWlC9DV40nkqs3Goe8X8tMPXn/HZn8Rz33703w8K/G6STnN0txhAT4tY7D3e0DA9UY87wNnpJ7dXoJqMXRv2dRgmUnGih/8cLHypyxBoLoL8qR9cWxAf/Cs+qQmsk15lzIGQUAJwwXBBjbnXKwykEeHjTHsvjd7zzC1cWtz5Zz/8aop7AsVwaBqb9u+5dVOMxdzLGD24NKTjhtG86ADU4Mpnlb5 >> >> >> >> rob > > Updated patch attached. > > Honza > Rebased patch attached. Honza -- Jan Cholasta -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-jcholast-83.2-openssh-style-public-keys.patch Type: text/x-patch Size: 34811 bytes Desc: not available URL: From jcholast at redhat.com Fri Sep 7 16:15:51 2012 From: jcholast at redhat.com (Jan Cholasta) Date: Fri, 07 Sep 2012 18:15:51 +0200 Subject: [Freeipa-devel] [PATCH] 1048 update certificate renewal scripts In-Reply-To: <5037F78F.5010509@redhat.com> References: <5037F78F.5010509@redhat.com> Message-ID: <504A1DB7.4000707@redhat.com> Dne 24.8.2012 23:52, Rob Crittenden napsal(a): > A couple of issues were found in the CA renewal scripts. The api wasn't > being initialized so restart_dirsrv() didn't have access to > api.env.startup_timeout() I believe it was I who mislead you into removing it when I reviewed the original CA renewal patch. Sorry :-) > > A cert was missing from our list of certs to translate into CS.cfg > directives. > > rob > ACK. Honza -- Jan Cholasta From rcritten at redhat.com Fri Sep 7 16:26:51 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 07 Sep 2012 12:26:51 -0400 Subject: [Freeipa-devel] [PATCH] 1048 update certificate renewal scripts In-Reply-To: <504A1DB7.4000707@redhat.com> References: <5037F78F.5010509@redhat.com> <504A1DB7.4000707@redhat.com> Message-ID: <504A204B.6090309@redhat.com> Jan Cholasta wrote: > Dne 24.8.2012 23:52, Rob Crittenden napsal(a): >> A couple of issues were found in the CA renewal scripts. The api wasn't >> being initialized so restart_dirsrv() didn't have access to >> api.env.startup_timeout() > > I believe it was I who mislead you into removing it when I reviewed the > original CA renewal patch. Sorry :-) > >> >> A cert was missing from our list of certs to translate into CS.cfg >> directives. >> >> rob >> > > ACK. > > Honza > pushed to master and ipa-3-0 From rcritten at redhat.com Fri Sep 7 16:31:42 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 07 Sep 2012 12:31:42 -0400 Subject: [Freeipa-devel] [PATCH] 83 Use OpenSSH-style public keys as the preferred format of SSH public keys In-Reply-To: <504A11B1.7050003@redhat.com> References: <50463229.5030305@redhat.com> <5047BCBE.1090304@redhat.com> <5048C5A0.5090204@redhat.com> <504A11B1.7050003@redhat.com> Message-ID: <504A216E.8080305@redhat.com> Jan Cholasta wrote: > Dne 6.9.2012 17:47, Jan Cholasta napsal(a): >> Dne 5.9.2012 22:57, Rob Crittenden napsal(a): >>> Jan Cholasta wrote: >>>> Hi, >>>> >>>> this patch changes the format of the sshpubkey parameter to the format >>>> used by OpenSSH (see sshd(8)). >>>> >>>> Public keys in the old format (raw RFC 4253 blob) are automatically >>>> converted to OpenSSH-style public keys. OpenSSH-style public keys are >>>> now stored in LDAP. >>>> >>>> Changed sshpubkeyfp to be an output parameter, as that is what it >>>> actually is. >>>> >>>> Allow parameter normalizers to be used on values of any type, not just >>>> unicode, so that public key blobs (which are str) can be normalized to >>>> OpenSSH-style public keys. >>>> >>>> Note that you need a SSSD build including >>>> >>>> >>>> >>>> >>>> (SSSD 1.9.0beta7-to-be) in order to make OpenSSH integration actually >>>> work with OpenSSH-style public keys. >>>> >>>> >>>> >>>> >>>> Honza >>> >>> NACK. >>> >>> I think a bunch of tests are needed for this. >>> >>> Because you abstracted out the pubkey class it should be straightforward >>> to add a bunch of class-based unit tests on it. >>> >>> There are also no user or host-based tests, either for adding or >>> managing keys. >> >> Tests added. >> >>> >>> I tested backwards compatibility with 2.2 and the initial tests are >>> mixed. >>> >>> I installed 2.2 and created a 3.0 clone from it, including your patch. >> >> Do people actually do that in real deployments? >> >>> >>> I added a user in 3.0 with a key and it added ok, but on the 2.2 side it >>> returns the entire base64 encoded blob of key type, key and comment, >>> which I presume is unusable. At least things don't blow up. >> >> The format of ipasshpubkey in LDAP has changed, so there's not much I >> can do about this. >> >>> >>> The reverse works fine. An old-style key added to 2.2 appears to work >>> fine in 3.0, we just lack a comment. >>> >>> On the 2.2 server: >>> >>> $ ipa user-show tuser1 --all | grep -i ssh >>> Base-64 encoded SSH public key: >>> c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDNUQyRTI2dHU5YXM2cHhlUVlSdUgzelYyUDUzMjFpR1U5aC9XNElpd0tGSGlOc2p5cXFyemhCUFB3am83dGlYRDlHbUo1M25KS21OTGd0K01XUnFTZEx2R0V3NjM3SkVTWEpGL0VWeUxvZEFWRGltdXFRVkNLWjBRcm1kYjErRUg1VGRrd3ByOExyd0g1a0RzMEVpcGc2c0xoRUZ5NzMvaXNjRkJqcmk0NGxSU1BZNXFHTWFLOVE0cjY1WFEyaytlZ1RDQnBNZnc0b0J6Mzh0ZHVEVVE2bW9XNFhQSnhZeWJ3MGFDMnRUK2RBOU42WndFSFZXREUzdzg0bHRHa0JRZFRaKzViRnBFdlladm9PbkZXdDlNZFIzYVd6UklnY1o5VDlySDFFT2Z3eE5zWVRCLzRjTmg3dS9adGxnMVV0Z1VteWN3TkpMTUYrMTNzNTl2OFFpSFogcmNyaXRAZWRzZWwuZ3JleW9hay5jb20= >>> >>> >>> >>> $ python >>> Python 2.7.3 (default, Jul 24 2012, 10:05:38) >>> [GCC 4.7.0 20120507 (Red Hat 4.7.0-5)] on linux2 >>> Type "help", "copyright", "credits" or "license" for more information. >>> >>> import base64 >>> >>> s = >>> 'c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDNUQyRTI2dHU5YXM2cHhlUVlSdUgzelYyUDUzMjFpR1U5aC9XNElpd0tGSGlOc2p5cXFyemhCUFB3am83dGlYRDlHbUo1M25KS21OTGd0K01XUnFTZEx2R0V3NjM3SkVTWEpGL0VWeUxvZEFWRGltdXFRVkNLWjBRcm1kYjErRUg1VGRrd3ByOExyd0g1a0RzMEVpcGc2c0xoRUZ5NzMvaXNjRkJqcmk0NGxSU1BZNXFHTWFLOVE0cjY1WFEyaytlZ1RDQnBNZnc0b0J6Mzh0ZHVEVVE2bW9XNFhQSnhZeWJ3MGFDMnRUK2RBOU42WndFSFZXREUzdzg0bHRHa0JRZFRaKzViRnBFdlladm9PbkZXdDlNZFIzYVd6UklnY1o5VDlySDFFT2Z3eE5zWVRCLzRjTmg3dS9adGxnMVV0Z1VteWN3TkpMTUYrMTNzNTl2OFFpSFogcmNyaXRAZWRzZWwuZ3JleW9hay5jb20=' >>> >>> >>> >>> >>> base64.b64decode(s) >>> 'ssh-rsa >>> AAAAB3NzaC1yc2EAAAADAQABAAABAQC5D2E26tu9as6pxeQYRuH3zV2P5321iGU9h/W4IiwKFHiNsjyqqrzhBPPwjo7tiXD9GmJ53nJKmNLgt+MWRqSdLvGEw637JESXJF/EVyLodAVDimuqQVCKZ0Qrmdb1+EH5Tdkwpr8LrwH5kDs0Eipg6sLhEFy73/iscFBjri44lRSPY5qGMaK9Q4r65XQ2k+egTCBpMfw4oBz38tduDUQ6moW4XPJxYybw0aC2tT+dA9N6ZwEHVWDE3w84ltGkBQdTZ+5bFpEvYZvoOnFWt9MdR3aWzRIgcZ9T9rH1EOfwxNsYTB/4cNh7u/Ztlg1UtgUmycwNJLMF+13s59v8QiHZ >>> >>> >>> rcrit at edsel.greyoak.com' >>> >>> Now show an old style key: >>> >>> $ ipa user-show tuser2 --all | grep -i ssh >>> Base-64 encoded SSH public key: >>> AAAAB3NzaC1yc2EAAAADAQABAAABAQCbRLyizFGyfucNRnHpWdUG8dBD7W2PfvTQ42k+LmAdUFudTytO89oTRXcVEYMDL42OyRth12JRMUjYTEmFwo9a9Mb7cP8+bo7N2lV4iCB0CUybcZARF0MV6NeYhhWlC9DV40nkqs3Goe8X8tMPXn/HZn8Rz33703w8K/G6STnN0txhAT4tY7D3e0DA9UY87wNnpJ7dXoJqMXRv2dRgmUnGih/8cLHypyxBoLoL8qR9cWxAf/Cs+qQmsk15lzIGQUAJwwXBBjbnXKwykEeHjTHsvjd7zzC1cWtz5Zz/8aop7AsVwaBqb9u+5dVOMxdzLGD24NKTjhtG86ADU4Mpnlb5 >>> >>> >>> >>> >>> rob >> >> Updated patch attached. >> >> Honza >> > > Rebased patch attached. > > Honza > ACK. I merged in a change that adds e-mail to one test and pushed to master and ipa-3-0. rob From rcritten at redhat.com Fri Sep 7 16:35:02 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 07 Sep 2012 12:35:02 -0400 Subject: [Freeipa-devel] [PATCH] 205 Reflect API change of SSH store in Web UI In-Reply-To: <5047C5B9.3080809@redhat.com> References: <504709CF.4010201@redhat.com> <5047C5B9.3080809@redhat.com> Message-ID: <504A2236.7060907@redhat.com> Endi Sukma Dewata wrote: > On 9/5/2012 3:14 AM, Petr Vobornik wrote: >> Format of ipasshpubkey in users and hosts changed from BYTES to STR. Web >> UI no longer gets the value as base64 encoded string in an object. >> >> Label was changed to reflect that the key don't have to be plain base64 >> encoded blob. >> >> https://fedorahosted.org/freeipa/ticket/2989 >> >> Note: freeipa-jcholast-83-openssh-style-public-keys.patch should be >> applied > > ACK. > > Possible improvements: > > 1. Right now if you click 'Add' SSH public key you'd have to click > 'Show/Set key' to enter the value. We probably could make it such that > when you click 'Add' it will open the input dialog immediately. This way > we can avoid an incomplete state where a slot for a new key is added but > it's empty. > > 2. If we do #1 the 'New: key set/not set' label can be changed to 'New > key'. The 'Modified' can be changed to 'Modified key'. > > 3. The 'Show/Set key' probably can be changed to 'View/Edit' to be more > consistent with host/service certificate. > pushed to ipa-3-0 and master rob From rmeggins at redhat.com Fri Sep 7 19:00:03 2012 From: rmeggins at redhat.com (Rich Megginson) Date: Fri, 07 Sep 2012 13:00:03 -0600 Subject: [Freeipa-devel] 389-ds-base-1.2.11.14 is available in testing Message-ID: <504A4433.9090607@redhat.com> This should fix the problem with CLEANALLRUV and winsync agreements From pviktori at redhat.com Mon Sep 10 14:12:25 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Mon, 10 Sep 2012 16:12:25 +0200 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <50489B48.70503@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502B86F5.4070008@redhat.com> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> Message-ID: <504DF549.5040007@redhat.com> On 09/06/2012 02:47 PM, Petr Viktorin wrote: > On 09/06/2012 02:35 PM, Rob Crittenden wrote: >> Ade Lee wrote: >>> On Wed, 2012-09-05 at 16:20 -0400, Rob Crittenden wrote: >>>> Martin Kosek wrote: >>>>> On 08/31/2012 04:53 PM, Petr Viktorin wrote: >>>>>> On 08/28/2012 03:40 PM, Petr Viktorin wrote: >>>>>>> On 08/17/2012 06:04 PM, Ade Lee wrote: >>>>>>>> On Fri, 2012-08-17 at 09:34 -0400, Ade Lee wrote: >>>>>>>>> On Thu, 2012-08-16 at 18:45 +0200, Martin Kosek wrote: >>>>>>>>>> On 08/16/2012 01:28 PM, Ade Lee wrote: >>>>>>>>>>> Patch attached this time. I should know better than to do >>>>>>>>>>> this in the >>>>>>>>>>> middle of the night .. >>>>>>>>>>> >>>>>>>>>>> On Thu, 2012-08-16 at 09:12 +0200, Martin Kosek wrote: >>>>>>>>>>>> On 08/16/2012 07:53 AM, Ade Lee wrote: >>>>>>>>>>>>> On Wed, 2012-08-15 at 23:41 -0400, Ade Lee wrote: >>>>>>>>>>>>>> On Wed, 2012-08-15 at 16:34 +0200, Martin Kosek wrote: >>>>>>>>>>>>>>> On 08/15/2012 03:54 PM, Ade Lee wrote: >>>>>>>>>>>>>>>> On Wed, 2012-08-15 at 13:24 +0200, Martin Kosek wrote: >>>>>>>>>>>>>>>>> On 08/08/2012 10:05 PM, Ade Lee wrote: >>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Dogtag 10 is being released on f18, and has a number of >>>>>>>>>>>>>>>>>> changes that >>>>>>>>>>>>>>>>>> will affect IPA. In particular, the following changes >>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>> affect >>>>>>>>>>>>>>>>>> current IPA code. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> * The directory layout of the dogtag instance has >>>>>>>>>>>>>>>>>> changed. >>>>>>>>>>>>>>>>>> Instead of >>>>>>>>>>>>>>>>>> using separate tomcat instances to host different >>>>>>>>>>>>>>>>>> subsystems, the >>>>>>>>>>>>>>>>>> standard dogtag installation will allow one to install >>>>>>>>>>>>>>>>>> a CA. >>>>>>>>>>>>>>>>>> KRA, OCSP >>>>>>>>>>>>>>>>>> and TKS within the same instance. There have been >>>>>>>>>>>>>>>>>> corresponding changes >>>>>>>>>>>>>>>>>> in the directory layout, as well as the default >>>>>>>>>>>>>>>>>> instance name >>>>>>>>>>>>>>>>>> (pki-tomcat instead of pki-ca), and startup daemon >>>>>>>>>>>>>>>>>> (pki-tomcatd, instead >>>>>>>>>>>>>>>>>> of pki-cad, pki-krad etc.) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> * The default instance will use only four ports (HTTPS, >>>>>>>>>>>>>>>>>> HTTP, AJP and >>>>>>>>>>>>>>>>>> tomcat shutdown port) rather than the 6 previously used. >>>>>>>>>>>>>>>>>> The default >>>>>>>>>>>>>>>>>> ports will be changed to the standard tomcat ports. As >>>>>>>>>>>>>>>>>> these ports are >>>>>>>>>>>>>>>>>> local to the ipa server machine, this should not cause >>>>>>>>>>>>>>>>>> too much >>>>>>>>>>>>>>>>>> disruption. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> * There is a new single step installer written in python. >>>>>>>>>>>>>>>>>> (pkispawn/destroy) vs. pkicreate/pkisilent/pkiremove. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> * Dogtag 10 runs on tomcat7 - with a new corresponding >>>>>>>>>>>>>>>>>> version of >>>>>>>>>>>>>>>>>> tomcatjss. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The attached patch integrates all the above changes in >>>>>>>>>>>>>>>>>> IPA >>>>>>>>>>>>>>>>>> installation >>>>>>>>>>>>>>>>>> and maintenance code. Once the patch is applied, users >>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>> be able to: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 1. run ipa-server-install to completion on f18 with >>>>>>>>>>>>>>>>>> dogtag 10. >>>>>>>>>>>>>>>>>> 2. install a new replica on f18 on dogtag 10. >>>>>>>>>>>>>>>>>> 3. upgrade an f17 machine with an existing IPA >>>>>>>>>>>>>>>>>> instance to >>>>>>>>>>>>>>>>>> f18/ dogtag >>>>>>>>>>>>>>>>>> 10 - and have that old-style dogtag instance continue >>>>>>>>>>>>>>>>>> to run >>>>>>>>>>>>>>>>>> correctly. >>>>>>>>>>>>>>>>>> This will require the installation of the latest >>>>>>>>>>>>>>>>>> version of >>>>>>>>>>>>>>>>>> tomcatjss as >>>>>>>>>>>>>>>>>> well as the installation of tomcat6. The old-style >>>>>>>>>>>>>>>>>> instance >>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>> continue to use tomcat6. >>>>>>>>>>>>>>>>>> 4. in addition, the new cert renewal code has been >>>>>>>>>>>>>>>>>> patched >>>>>>>>>>>>>>>>>> and should >>>>>>>>>>>>>>>>>> continue to work. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> What is not yet completed / supported: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 1. Installation with an external CA is not yet >>>>>>>>>>>>>>>>>> completed in >>>>>>>>>>>>>>>>>> the new >>>>>>>>>>>>>>>>>> installer. We plan to complete this soon. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 2. There is some IPA upgrade code that has not yet been >>>>>>>>>>>>>>>>>> touched >>>>>>>>>>>>>>>>>> (install/tools/ipa-upgradeconfig). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 3. A script needs to be written to allow admins to >>>>>>>>>>>>>>>>>> convert >>>>>>>>>>>>>>>>>> their >>>>>>>>>>>>>>>>>> old-style dogtag instances to new style instances, as >>>>>>>>>>>>>>>>>> well >>>>>>>>>>>>>>>>>> as code to >>>>>>>>>>>>>>>>>> periodically prompt admins to do this. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 4. Installation of old-style instances using >>>>>>>>>>>>>>>>>> pkicreate/pkisilent on >>>>>>>>>>>>>>>>>> dogtag 10 will no longer be supported, and will be >>>>>>>>>>>>>>>>>> disabled >>>>>>>>>>>>>>>>>> soon. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 5. The pki-selinux policy has been updated to reflect >>>>>>>>>>>>>>>>>> these >>>>>>>>>>>>>>>>>> changes, >>>>>>>>>>>>>>>>>> but is still in flux. In fact, it is our intention to >>>>>>>>>>>>>>>>>> place >>>>>>>>>>>>>>>>>> the dogtag >>>>>>>>>>>>>>>>>> selinux policy in the base selinux policy for f18. In >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> meantime, it >>>>>>>>>>>>>>>>>> may be necessary to run installs in permissive mode. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The dogtag 10 code will be released shortly into f18. >>>>>>>>>>>>>>>>>> Prior >>>>>>>>>>>>>>>>>> to that >>>>>>>>>>>>>>>>>> though, we have placed the new dogtag 10 and tomcatjss >>>>>>>>>>>>>>>>>> code >>>>>>>>>>>>>>>>>> in a >>>>>>>>>>>>>>>>>> developer repo that is located at >>>>>>>>>>>>>>>>>> http://nkinder.fedorapeople.org/dogtag-devel/ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Testing can be done on both f18 and f17 - although the >>>>>>>>>>>>>>>>>> target platform - >>>>>>>>>>>>>>>>>> and the only platform for which official builds will be >>>>>>>>>>>>>>>>>> created is f18. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>> Ade >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Ade, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks for the patch, I started with review and >>>>>>>>>>>>>>>>> integration >>>>>>>>>>>>>>>>> tests (currently >>>>>>>>>>>>>>>>> running on Fedora 17 with Nathan's repo). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Installation on single master was smooth, it worked just >>>>>>>>>>>>>>>>> fine, even with >>>>>>>>>>>>>>>>> enforced SELinux, without any error - kudos to you and the >>>>>>>>>>>>>>>>> whole dogtag team. >>>>>>>>>>>>>>>>> The resulting logs and the structure of your log directory >>>>>>>>>>>>>>>>> seems improved. I >>>>>>>>>>>>>>>>> believe that the brand new Python installers will make it >>>>>>>>>>>>>>>>> easier to debug >>>>>>>>>>>>>>>>> issues with dogtag installation when they come. When I >>>>>>>>>>>>>>>>> tried >>>>>>>>>>>>>>>>> our unit tests or >>>>>>>>>>>>>>>>> some simple cert operation, it worked fine as well. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Now the bad news, or rather few issues and suggestions I >>>>>>>>>>>>>>>>> found: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1) As we already discussed on IRC, tomcat 7 was not pulled >>>>>>>>>>>>>>>>> automatically on >>>>>>>>>>>>>>>>> Fedora 17 when I updated pki-ca, you somewhere miss a >>>>>>>>>>>>>>>>> Requires. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> We have a dogtag patch that is currently in review that >>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>> address >>>>>>>>>>>>>>>> this. Once this is in, tomcatjss >=7.0.0 will be >>>>>>>>>>>>>>>> required for >>>>>>>>>>>>>>>> f17+, >>>>>>>>>>>>>>>> rather than f18+ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2) I had installed IPA with dogtag10 on master. >>>>>>>>>>>>>>>>> However, CA >>>>>>>>>>>>>>>>> installation on a >>>>>>>>>>>>>>>>> replica (ipa-ca-install) with dogtag9 failed - is this >>>>>>>>>>>>>>>>> expectable? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Yes. The current IPA patch is designed to work with >>>>>>>>>>>>>>>> dogtag 10 >>>>>>>>>>>>>>>> only, >>>>>>>>>>>>>>>> which will be officially available on f18+. So if you >>>>>>>>>>>>>>>> update >>>>>>>>>>>>>>>> to dogtag >>>>>>>>>>>>>>>> 10, you must have this patch and visa versa. We probably >>>>>>>>>>>>>>>> need >>>>>>>>>>>>>>>> to add >>>>>>>>>>>>>>>> the relevant requires to IPA to enforce this. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> If you have an existing dogtag 9 instance, and you >>>>>>>>>>>>>>>> upgrade to >>>>>>>>>>>>>>>> the new >>>>>>>>>>>>>>>> dogtag 10 and patched IPA, then that instance will >>>>>>>>>>>>>>>> continue to >>>>>>>>>>>>>>>> work. >>>>>>>>>>>>>>>> But any new instances would be created using dogtag 10. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 3) I had installed IPA with dogtag10 on master. Replica >>>>>>>>>>>>>>>>> had >>>>>>>>>>>>>>>>> dogtag10 as well >>>>>>>>>>>>>>>>> and I got the following error: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> # ipa-ca-install >>>>>>>>>>>>>>>>> /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>>> Configuring certificate server: Estimated time 3 >>>>>>>>>>>>>>>>> minutes 30 >>>>>>>>>>>>>>>>> seconds >>>>>>>>>>>>>>>>> [1/14]: creating certificate server user >>>>>>>>>>>>>>>>> [2/14]: configuring certificate server instance >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Your system may be partly configured. >>>>>>>>>>>>>>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Unexpected error - see >>>>>>>>>>>>>>>>> /var/log/ipareplica-ca-install.log for >>>>>>>>>>>>>>>>> details: >>>>>>>>>>>>>>>>> IOError: [Errno 2] No such file or directory: >>>>>>>>>>>>>>>>> '/var/lib/pki/pki-tomcat/alias/ca_backup_keys.p12' >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Root cause: >>>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>>> File >>>>>>>>>>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> line >>>>>>>>>>>>>>>>> 625, in __spawn_instance >>>>>>>>>>>>>>>>> "/root/cacert.p12") >>>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I need to look into this. I had fixed ipa-replica-install, >>>>>>>>>>>>>>>> rather than >>>>>>>>>>>>>>>> ipa-ca-install to create replicas. I didn't know >>>>>>>>>>>>>>>> ipa-ca-install >>>>>>>>>>>>>>>> existed! Should not be too bad to fix though - most likely >>>>>>>>>>>>>>>> just need to >>>>>>>>>>>>>>>> move files to the right place. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Ok, thanks! Btw. CA on replica can be either installed >>>>>>>>>>>>>>> during >>>>>>>>>>>>>>> ipa-replica-install time (when --setup-ca option is >>>>>>>>>>>>>>> passed, you >>>>>>>>>>>>>>> probably used >>>>>>>>>>>>>>> that one) and the aforementioned ipa-ca-install run after >>>>>>>>>>>>>>> ipa-replica-install. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> I will be testing this out again. As ipa-ca-install uses the >>>>>>>>>>>>>> same code >>>>>>>>>>>>>> as ipa-replica-install, I would have expected it to work. >>>>>>>>>>>>>> Did >>>>>>>>>>>>>> you try >>>>>>>>>>>>>> it with selinux in permissive mode? >>>>>>>>>>>>>> >>>>>>>>>>>>> OK -- I tested this out and realized that between the time I >>>>>>>>>>>>> initially >>>>>>>>>>>>> tested this and your tests, we had changed a URL >>>>>>>>>>>>> from /ca/pki/installer/installToken to >>>>>>>>>>>>> /ca/rest/installer/installToken, >>>>>>>>>>>>> but I forgot to update ipa-pki-proxy.conf. >>>>>>>>>>>>> >>>>>>>>>>>>> The new patch has been rebased and changes the relevant >>>>>>>>>>>>> patch. With >>>>>>>>>>>>> this, the CA replica installs correctly. >>>>>>>>>>>> >>>>>>>>>>>> Umh, where can I find the new rebased patch? :-) >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> There was one issue that I encountered. I have seen this once >>>>>>>>>>>>> before >>>>>>>>>>>>> and will need to investigate further. IPA sometimes >>>>>>>>>>>>> restarts the >>>>>>>>>>>>> dogtag >>>>>>>>>>>>> instance by doing systemctl stop pki-tomcatd.target. and then >>>>>>>>>>>>> systemctl >>>>>>>>>>>>> start pki-tomcatd.target. I have noticed that occasionally >>>>>>>>>>>>> the >>>>>>>>>>>>> start >>>>>>>>>>>>> invocation fails, while the stop and restart invocations >>>>>>>>>>>>> succeed >>>>>>>>>>>>> just >>>>>>>>>>>>> fine. This makes absolutely no sense because restart is >>>>>>>>>>>>> essentially >>>>>>>>>>>>> just stop and then start. >>>>>>>>>>>>> >>>>>>>>>>>>> I think this is actually a bug in systemd. If I reboot the >>>>>>>>>>>>> machine, it >>>>>>>>>>>>> all works perfectly. If we can't figure out whats going on >>>>>>>>>>>>> with >>>>>>>>>>>>> systemd, we can always replace this start/stop with >>>>>>>>>>>>> starting/stopping >>>>>>>>>>>>> the service directly. (pki-tomcatd at pki-tomcat.service). That >>>>>>>>>>>>> seems to >>>>>>>>>>>>> work all the time with no problems. >>>>>>>>>>>>> >>>>>>>>>>>>> I suggest we leave this fix (if needed) for a separate patch. >>>>>>>>>>>> >>>>>>>>>>>> Ok, I will see if I hit this issue again. Btw. is it >>>>>>>>>>>> recommended >>>>>>>>>>>> in systemd to >>>>>>>>>>>> use target units this way? I thought that only service files >>>>>>>>>>>> are >>>>>>>>>>>> meant to be >>>>>>>>>>>> used for manipulation with a service. As you said yourself, >>>>>>>>>>>> using >>>>>>>>>>>> service unit >>>>>>>>>>>> file for restart worked every time. >>>>>>>>>>>> >>>>>>>>>>>> Martin >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Now, I tested the rebased patch with VMs with permissive SELinux. >>>>>>>>>> IPA server >>>>>>>>>> install was OK, as always, but replica installation ended up with >>>>>>>>>> error. >>>>>>>>>> Although it got much further than before: >>>>>>>>>> >>>>>>>>>> # ipa-ca-install >>>>>>>>>> /home/mkosek/replica-info-vm-114.idm.lab.bos.redhat.com.gpg >>>>>>>>>> ... >>>>>>>>>> [3/3]: restarting directory server >>>>>>>>>> done configuring pkids. >>>>>>>>>> Configuring certificate server: Estimated time 3 minutes 30 >>>>>>>>>> seconds >>>>>>>>>> [1/14]: creating certificate server user >>>>>>>>>> [2/14]: configuring certificate server instance >>>>>>>>>> [3/14]: disabling nonces >>>>>>>>>> [4/14]: importing CA chain to RA certificate database >>>>>>>>>> [5/14]: fixing RA database permissions >>>>>>>>>> [6/14]: setting up signing cert profile >>>>>>>>>> [7/14]: set up CRL publishing >>>>>>>>>> [8/14]: set certificate subject base >>>>>>>>>> [9/14]: enabling Subject Key Identifier >>>>>>>>>> [10/14]: configuring certificate server to start on boot >>>>>>>>>> [11/14]: configure certmonger for renewals >>>>>>>>>> [12/14]: configure clone certificate renewals >>>>>>>>>> [13/14]: configure Server-Cert certificate renewal >>>>>>>>>> [14/14]: Configure HTTP to proxy connections >>>>>>>>>> done configuring pki-tomcatd. >>>>>>>>>> Restarting the directory and certificate servers >>>>>>>>>> >>>>>>>>>> Your system may be partly configured. >>>>>>>>>> Run /usr/sbin/ipa-server-install --uninstall to clean up. >>>>>>>>>> >>>>>>>>>> It seems like that CA on a replica did not start and so a check >>>>>>>>>> for >>>>>>>>>> port 8080 >>>>>>>>>> failed. Attached logs (pki-ca-logs.tgz) may provide more >>>>>>>>>> information >>>>>>>>>> to this issue. >>>>>>>>>> >>>>>>>>> This is the systemd issue I mentioned before. I will submit a >>>>>>>>> patch >>>>>>>>> which will change the restart mechanism to restart the service >>>>>>>>> and not >>>>>>>>> the target. >>>>>>>>>> >>>>>>>> >>>>>>>> Patch attached. This patch should be applied on top of the large >>>>>>>> patch >>>>>>>> (being reviewed). >>>>>>>> >>>>>>>>>> Now I am also testing upgrade from IPA+dogtag9 to >>>>>>>>>> PatchedIPA+dogtag10 >>>>>>>>>> (permissive SELinux). Now, the service was upgraded smoothly, CA >>>>>>>>>> service >>>>>>>>>> could be restarted without failure. However, certificate >>>>>>>>>> operations now >>>>>>>>>> did not work: >>>>>>>>>> >>>>>>>>>> # ipactl restart >>>>>>>>>> Restarting Directory Service >>>>>>>>>> Restarting KDC Service >>>>>>>>>> Restarting KPASSWD Service >>>>>>>>>> Restarting MEMCACHE Service >>>>>>>>>> Restarting HTTP Service >>>>>>>>>> Restarting CA Service >>>>>>>>>> # ipactl status >>>>>>>>>> Directory Service: RUNNING >>>>>>>>>> KDC Service: RUNNING >>>>>>>>>> KPASSWD Service: RUNNING >>>>>>>>>> MEMCACHE Service: RUNNING >>>>>>>>>> HTTP Service: RUNNING >>>>>>>>>> CA Service: RUNNING >>>>>>>>>> # ipa cert-show 1 >>>>>>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to >>>>>>>>>> communicate with CMS (Internal Server Error) >>>>>>>>>> >>>>>>>>>> Attaching logs for this case as well (ipa-ca-logs-upgrade.tgz). >>>>>>>>>> >>>>>>>>> The reason for this is that the old dogtag instances are missing: >>>>>>>>> 1. a new parameter in CS.cfg >>>>>>>>> 2. a couple of symlinks >>>>>>>>> >>>>>>>>> I plan to fix this in the dogtag code. Specifically, >>>>>>>>> 1. I will modify the dogtag code to provide a default value in >>>>>>>>> the case >>>>>>>>> where the parameter does not exist. >>>>>>>>> 2. I plan to add a function to the dogtag startup scripts to >>>>>>>>> check and >>>>>>>>> remake any required symlinks. >>>>>>>>> >>>>>>>>> Both of these changes are in the dogtag code, and do not affect >>>>>>>>> this >>>>>>>>> patch. As a workaround, to confirm that the upgrade actually >>>>>>>>> works, do >>>>>>>>> the following manual steps on your upgraded instance: >>>>>>>>> >>>>>>>>> 1. Add the following parameter to CS.cfg >>>>>>>>> pidDir=/var/run >>>>>>>>> >>>>>>>>> 2. Add the following links; >>>>>>>>> cd /var/lib/pki-ca/common/lib >>>>>>>>> ln -s /usr/share/java/apache-commons-codec.jar >>>>>>>>> apache-commons-codec.jar >>>>>>>>> ln -s /usr/share/java/log4j.jar log4j.jar >>>>>>>>> >>>>>>>>> 3 Restart the dogtag instance >>>>>>>>> >>>>>>>>>> HTH, >>>>>>>>>> Martin >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> The attached patch conditionalizes the changes, so that IPA supports >>>>>>> both Dogtag versions. >>>>>>> >>>>>>> I didn't put it in platform code for two reasons >>>>>>> - One platform (f17) can have either Dogtag version installed >>>>>>> - I didn't want to conflict with Timo Aaltonen's "platformizing" >>>>>>> work. >>>>>>> According to his WIP patches, he plans to move the whole dogtag >>>>>>> module >>>>>>> into platform code. >>>>>>> >>>>>>> I used the existence of /usr/bin/pkispawn as a test for Dogtag >>>>>>> 10. If >>>>>>> you know a better way, please comment. >>>>>>> >>>>>>> The dogtag_version option is now always added to >>>>>>> >>>>>>> I've reverted the changes to install/ui/test/data/ipa_init.json, so >>>>>>> you'll have to change the ports manually when testing the UI with >>>>>>> Dogtag >>>>>>> 10. >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> Attaching a patch with fixed ipa-pki-proxy.conf, as discussed on IRC. >>>>>> >>>>> >>>>> This approach looks good. I did not hit any regression on F17 with >>>>> dogtag9, or >>>>> clean installs of IPA+dogtag10. I think we are getting close to >>>>> pushing this code. >>>>> >>>>> The only issues I hit so far are for upgraded dogtag9 instances (on >>>>> F17): >>>>> >>>>> 1) The service did not start for me. There were some SELinux AVCs >>>>> and even when >>>>> I disabled SELinux the instance still did not start (logs attached). >>>>> >>>>> 2) Uninstall was also not clean, we leave some dogtag installation >>>>> states for >>>>> upgraded dogtag9 instance: >>>>> >>>>> # ipa-server-install --uninstall --unattended >>>>> Shutting down all IPA services >>>>> Removing IPA client configuration >>>>> Unconfiguring ntpd >>>>> Unconfiguring CA directory server >>>>> Unconfiguring web server >>>>> Unconfiguring krb5kdc >>>>> Unconfiguring kadmin >>>>> Unconfiguring directory server >>>>> Unconfiguring ipa_memcached >>>>> ipa : ERROR Some installation state for pki-cad has not >>>>> been >>>>> restored, see /var/lib/ipa/sysrestore/sysrestore.state >>>>> >>>>> /var/lib/ipa/sysrestore/sysrestore.state: >>>>> [pki-cad] >>>>> enabled = False >>>> >>>> Ade is working on a new build to address the dogtag upgrade issues. >>>> >>>> The left over state should be removed when we drop the separate >>>> instance >>>> in the dogtag 9 -> 10 upgrade. I'm not completely sure reading the >>>> patch >>>> who actually does this part, is this handled by dogtag? >>>> >>>> rob >>>> >>> >>> The build is available on the developer repo. Just do a yum update. >>> >>> As for this leftover state, this is an ipa file and such is never >>> handled by dogtag. It must be handled somewhere within the ipa code. >> >> Well, the question is, what handles the dogtag 9 -> dogtag 10 upgrade? >> >> If IPA isn't involved then it has no chance to remove this state. >> >> rob >> > > I think the problem is that I treated uninstall as part of the > installation code, so it's not looking at the configuration and assuming > a dogtag 10-style layout just because the dogtag 10 RPM is installed > (but in reality it's dogtag 10 managing the dogtag 9-style layout). > I'll look into it soon. > Sending Ade's patches rebased & squashed into one, my rebased patch, and a WIP of a fix for the uninstaller & ipactl. Please apply in that order. -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0074-03-Use-Dogtag-10-only-when-it-is-available.patch Type: text/x-patch Size: 49620 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-WIP-Support-upgraded-instance-of-Dogtag-9-10.patch Type: text/x-patch Size: 4456 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: rebased-squashed-alee-0002-Modifications-to-install-scripts-for-dogtag-10.patch Type: text/x-patch Size: 44489 bytes Desc: not available URL: From rcritten at redhat.com Mon Sep 10 15:02:44 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 10 Sep 2012 11:02:44 -0400 Subject: [Freeipa-devel] Announcing FreeIPA v3.0.0 beta 3 Message-ID: <504E0114.9090901@redhat.com> The FreeIPA team is proud to announce version FreeIPA v3.0.0 beta 3. It can be downloaded from http://www.freeipa.org/page/Downloads. A build is available only for Fedora 17 via the freeipa-devel repo on www.freeipa.org: http://freeipa.org/downloads/freeipa-devel.repo . To install in Fedora 17 the updates repo repository needs to be enabled as well. For additional information see the AD Trust design page http://freeipa.org/page/IPAv3_AD_trust and the AD Trust testing page http://freeipa.org/page/IPAv3_testing_AD_trust. == Highlights since 3.0.0 beta 2 == * Cooperate with new 389-ds-base winsync POSIX plugin so that AD POSIX attribute can be synced with IPA. * Improvements to schema upgrade process. * Prevent last admin from being disabled. * Exclude some attributes from replication. * Notify success on add, delete and update in UI. * Set the e-mail attribute on new users by default. * Rename range commands to idrange, * Improvements to idrange command-line. * SSH public key format has been changed to OpenSSH-style public keys. == Upgrading == An IPA server can be upgraded simply by installing updated rpms. The server does not need to be shut down in advance. If you have multiple servers you may upgrade them one at a time. It is expected that all servers will be upgraded in a relatively short period (days or weeks not months). They should be able to co-exist peacefully but new features will not be available on old servers and enrolling a new client against an old server will result in the SSH keys not being uploaded. Downgrading a server once upgraded is not supported. Upgrading from 2.2.0 should work but has not been fully tested. Proceed with caution. An enrolled client does not need the new packages installed unless you want to re-enroll it. SSH keys for already installed clients are not uploaded, you will have to re-enroll the client or manually upload the keys. == Feedback == Please provide comments, bugs and other feedback via the freeipa-devel mailing list: http://www.redhat.com/mailman/listinfo/freeipa-devel == Detailed changelog == Alexander Bokovoy (4): * Recover from invalid cached kerberos credentials in ipasam * Fix ipasam ipaNThash magic regen to actually fetch updated password * Add ACI to allow regenerating ipaNTHash from ipasam * Ask for admin password in ipa-adtrust-install Jan Cholasta (1): * Use OpenSSH-style public keys as the preferred format of SSH public keys. John Dennis (4): * DN objects hash differently depending on case * ipactl exception not handled well * ipa user-find --manager does not find matches * prevent last admin from being disabled Martin Kosek (12): * Read DM password from option in external CA install * Fix client-only build * Fix managedBy label for DNS zone * Update Contributors.txt file * Make replica install more robust * Add safe updates for objectClasses * Allow localhost in zone ACIs * Transfer long numbers over XMLRPC * Fix DNS SOA serial parameters boundaries * Add range safety check for range_mod and range_del * Update DNS zone allow-query validation test * Cast DNS SOA serial maximum boundary to long Petr Viktorin (3): * Internationalization for public errors * Run ntpdate in verbose mode, not debug (i.e. no-op) mode * Add nsds5ReplicaStripAttrs to replica agreements Petr Vobornik (15): * Range Web UI * Revert change causing failure in test automation * Fix issue which broke setup of Web UI unit tests * Successful action notification * Password policy paging with proper sorting * Fixed search in HBAC test * Permissions: select only applicable options on type change * Notify success on add, delete and update * Fixed metadata serialization of Numbers and DNs * Added decimal checks to metadata validator * Generated metadata for testing updated * Fixed problem while deleting entry with unsaved changes * Allow localhost in zone ACIs - Web UI * Update of confirmation of actions * Reflect API change of SSH store in Web UI Rob Crittenden (8): * Don't generate password history error if history is set to 0. * Restrict the SELinux user map user MLS value to 0-1023 * Support the new Winsync POSIX API. * Set minimum of 389-ds-base to 1.2.11.8 to pick up cache warning. * Add version to replica prepare file, prevent installing to older version * Set the e-mail attribute using the default domain name by default * Fix some restart script issues found with certificate renewal. * Become IPA v3 beta 3 (3.0.0.pre3) Sumit Bose (27): * Use libsamba-security instead of libsecurity * ipadb_iterate(): handle match_entry == NULL * ipasam: cleanup explicit dependencies to samba libs * Make encode_ntlm_keys() public * ipasam: remove nt_lm_owf_gen() and dependency to libcliauth.so * ipasam: remove sid_peek_rid() * ipasam: replace strnequal() * ipasam: remove strlower_m() * ipasam: remove talloc_asprintf_strupper_m() * ipasam: replace sid_copy() * ipasam: replace sid_compose() * ipasam: Replace is_null_sid() * ipasam: Replace dom_sid_compare_domain() * ipasam: Replace sid_check_is_our_sam() * ipasam: Replace sid_peek_check_rid() * ipasam: Replace global_sid_Builtin * ipasam: add libsss_idmap context and replace string_to_sid() * ipasam: replace get_global_sam_sid() * ipasam: remove fetch_ldap_pw() * ipasam: replace trim_char() with trim_string() * Rename range CLI to idrange * IDRange CLI: allow to work without arguments * IDRange CLI: Add documentation * Do not create trust if murmur hash is not available and base-id not given * Trust CLI: Return more details when searching trusts * Trust CLI: return more details of added trust * Trust CLI: mark trust-mod for future use Tomas Babej (5): * Adds dependency on samba4-winbind. * Improves deletion of PTR records in ipa host-del * Fixes different behaviour of permission-mod and show. * Change slapi_mods_init in ipa_winsync_pre_ad_mod_user_mods_cb * Sort policies numerically in pwpolicy-find From simo at redhat.com Mon Sep 10 15:43:16 2012 From: simo at redhat.com (Simo Sorce) Date: Mon, 10 Sep 2012 11:43:16 -0400 Subject: [Freeipa-devel] [Freeipa-users] errors when one ipa server down In-Reply-To: <504E030A.7000503@redhat.com> References: <05E4D95B-464F-4872-BA92-CA4B27488C37@gmail.com> <504A1D65.40500@redhat.com> <199A95F6-FB79-4CAE-8BC7-09D0AD6B6E2B@gmail.com> <504A415B.1070609@redhat.com> <504A5DFA.1040401@redhat.com> <504B5E46.1020002@redhat.com> <504DE637.4010803@redhat.com> <20120910132051.GL28901@zeppelin.brq.redhat.com> <1347286046.1150.7325.camel@willson.li.ssimo.org> <20120910143651.GJ30746@localhost.localdomain> <1347289743.1150.7389.camel@willson.li.ssimo.org> <504E030A.7000503@redhat.com> Message-ID: <1347291796.1150.7427.camel@willson.li.ssimo.org> On Mon, 2012-09-10 at 11:11 -0400, Rob Crittenden wrote: > Simo Sorce wrote: > > On Mon, 2012-09-10 at 16:36 +0200, Sumit Bose wrote: > >> What about defining a task in the SSSD krb5 provider instead of > >> pinging > >> it from the locator plugin. The task can run at a configurable > >> interval > >> or never and checks if the current KDC is available. If not it tries > >> the > >> next until it goes offline if no reachable KDC can be found and > >> updates > >> or deletes the info file for the locator plugin.. > >> > >> This leave us with the question how to ping a KDC properly, but this > >> we > >> have to find out for either case. > >> > > I am not a fan of generating load for the KDC unnecessarily. > > > > Simo. > > > > I tend to agree but this can be a real pain to debug because depending > on the current state of sssd you have to either check krb5.conf or the > sssd locator to see what KDC is configured. [moving to freeipa-devel] Yes but the solution is to do on-demand requests when something doesn't work. Because otherwise you still get the odd failure. Assume you check in 5 min intervals, and the KDC goes off 1 sec after the check, for 5 minutes you still have a wrong KDC in the locator and still get failures. So you loaded the KDC with ~300 request per day per client, and you still have high odds that on failure your locator file will still be 'wrong'. Simo. -- Simo Sorce * Red Hat, Inc * New York From pviktori at redhat.com Mon Sep 10 17:07:17 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Mon, 10 Sep 2012 19:07:17 +0200 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <504DF549.5040007@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502B86F5.4070008@redhat.com> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> Message-ID: <504E1E45.5090608@redhat.com> Attaching rebased and squashed patches. I've done some testing with them but please test some more. -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0074-04-Use-Dogtag-10-only-when-it-is-available.patch Type: text/x-patch Size: 52818 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: rebased--freeipa-alee-0001-Modifications-to-install-scripts-for-dogtag-10.patch Type: text/x-patch Size: 44489 bytes Desc: not available URL: From rcritten at redhat.com Mon Sep 10 18:34:05 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 10 Sep 2012 14:34:05 -0400 Subject: [Freeipa-devel] [PATCH] 1050 prevent replica orphans In-Reply-To: <1347008704.2829.13.camel@priserak> References: <5040F6E2.7060109@redhat.com> <5040F722.4060208@redhat.com> <5048BFAD.1070807@redhat.com> <50491414.2070504@redhat.com> <1347008704.2829.13.camel@priserak> Message-ID: <504E329D.6070905@redhat.com> Martin Kosek wrote: > On Thu, 2012-09-06 at 17:22 -0400, Rob Crittenden wrote: >> Martin Kosek wrote: >>> On 08/31/2012 07:40 PM, Rob Crittenden wrote: >>>> Rob Crittenden wrote: >>>>> It was possible use ipa-replica-manage connect/disconnect/del to end up >>>>> orphaning or or more IPA masters. This is an attempt to catch and >>>>> prevent that case. >>>>> >>>>> I tested with this topology, trying to delete B. >>>>> >>>>> A <-> B <-> C >>>>> >>>>> I got here by creating B and C from A, connecting B to C then deleting >>>>> the link from A to B, so it went from A -> B and A -> C to the above. >>>>> >>>>> What I do is look up the servers that the delete candidate host has >>>>> connections to and see if we're the last link. >>>>> >>>>> I added an escape clause if there are only two masters. >>>>> >>>>> rob >>>> >>>> Oh, this relies on my cleanruv patch 1031. >>>> >>>> rob >>>> >>> >>> 1) When I run ipa-replica-manage del --force to an already uninstalled host, >>> the new code will prevent me the deletation because it cannot connect to it. It >>> also crashes with UnboundLocalError: >>> >>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>> >>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing removal >>> Traceback (most recent call last): >>> File "/sbin/ipa-replica-manage", line 708, in >>> main() >>> File "/sbin/ipa-replica-manage", line 677, in main >>> del_master(realm, args[1], options) >>> File "/sbin/ipa-replica-manage", line 476, in del_master >>> sys.exit("Failed read master data from '%s': %s" % (delrepl.hostname, str(e))) >>> UnboundLocalError: local variable 'delrepl' referenced before assignment >> >> Fixed. >> >>> >>> >>> I also hit this error when removing a winsync replica. >> >> Fixed. >> >>> >>> >>> 2) As I wrote before, I think having --force option override the user inquiries >>> would benefit test automation: >>> >>> + if not ipautil.user_input("Continue to delete?", False): >>> + sys.exit("Aborted") >> >> Fixed. >> >>> >>> >>> 3) I don't think this code won't cover this topology: >>> >>> A - B - C - D - E >>> >>> It would allow you deleting a replica C even though it would separate A-B and >>> D-E. Though we may not want to cover this situation now, what you got is >>> definitely helping. >> >> I think you may be right. I only tested with 4 servers. With this B and >> D would both still have 2 agreements so wouldn't be covered by the last >> link test. > > Everything looks good now, so ACK. We just need to push it along with > CLEANALLRUV patch. > > Martin > Not to look a gift ACK In the mouth but here is a revised patch. I've added a cleanup routine to remove an orphaned master properly. If you had tried the mechanism I outlined in the man page it would have worked but was less-than-complete. This way is better, just don't try it on a live master. I also added a cleanruv abort command, in case you want to kill an existing task. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-rcrit-1050-3-replicaorphan.patch Type: text/x-diff Size: 12789 bytes Desc: not available URL: From edewata at redhat.com Mon Sep 10 19:12:03 2012 From: edewata at redhat.com (Endi Sukma Dewata) Date: Mon, 10 Sep 2012 14:12:03 -0500 Subject: [Freeipa-devel] [PATCH] 304 Allow localhost in zone ACIs In-Reply-To: <5048904F.6060802@redhat.com> References: <50487215.70600@redhat.com> <50488A8E.4030409@redhat.com> <5048904F.6060802@redhat.com> Message-ID: <504E3B83.4020308@redhat.com> On 9/6/2012 7:00 AM, Martin Kosek wrote: > On 09/06/2012 01:35 PM, Petr Vobornik wrote: >> On 09/06/2012 11:51 AM, Martin Kosek wrote: >>> Loopback address, "localhost" and "localnets" ACIs are no longer >>> an issue for bind-dyndb-ldap. Allow them in our validators. >>> >> >> Martin's patch works and looks good - ACK. >> >> >> Attaching patch for Web UI part. > > Web UI validator works fine too, ACK. > > Pushed both patches to master, ipa-3-0. > > Martin Looks like there's a problem in DNS zone details page. The idnssoaserial's maxvalue is an empty string so the validation fails even if you are just opening an existing DNS zone. So either the server should not return maxvalue or the UI should check for possible empty string. -- Endi S. Dewata From edewata at redhat.com Mon Sep 10 19:12:06 2012 From: edewata at redhat.com (Endi Sukma Dewata) Date: Mon, 10 Sep 2012 14:12:06 -0500 Subject: [Freeipa-devel] [PATCH] 211 Prevent opening of multiple dirty dialogs on navigation In-Reply-To: <5049EA77.9010308@redhat.com> References: <5049EA77.9010308@redhat.com> Message-ID: <504E3B86.1040204@redhat.com> On 9/7/2012 7:37 AM, Petr Vobornik wrote: > Facets which performs AJAX call after update refresh (clear dirty state) > after calling callback of dirty dialog. It might lead to multiple > openings of dirty dialog. > > Assuming that calling dirty dialog's callback can be evaluated as "dirty > state is gone", we can call reset in the callback to prevent the issue. > There will be an incorrect state in the facet for a moment. It will be > fixed soon on execute of callback of the refresh AJAX call. It is not an > issue because it will happen in background. User will be looking on > different facet. > > https://fedorahosted.org/freeipa/ticket/2667 ACK. -- Endi S. Dewata From rcritten at redhat.com Mon Sep 10 20:58:40 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 10 Sep 2012 16:58:40 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <504E1E45.5090608@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502B86F5.4070008@redhat.com> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> Message-ID: <504E5480.3000201@redhat.com> Petr Viktorin wrote: > Attaching rebased and squashed patches. I've done some testing with them > but please test some more. > Most of these aren't IPA issues, but dogtag issues. I'll try to split them out. IPA: For the configuration files in install/conf to be updated at rpm update time the VERSION needs to be incremented. The ipa package lacks any updated dogtag dependencies, so I abused it. I installed IPA with dogtag 9 and created a replica. I updated the IPA bits, that worked fine. I updated to dogtag 10 and now the CA doesn't work on the master, including starting the dogtag instance. Note that the rpm update process worked, no notice that the CA service didn't restart. Uninstalling failed because it tried to run pkidestroy and not pkiremove. The contents of the file passed to pkispawn should be logged so we can see exactly what was passed in. DOGTAG: When upgrading using the dogtag-devel repo I had to specify pki-tools.x86_64 otherwise it tried to install both 32 and 64-bit versions (and failed). I ended up running: yum update pki-ca tomcatjss pki-tools.x86_64 --enablerepo=dogtag-devel --enablerepo=updates-testing What happens if someone manually upgrades pki-ca without first updating ipa? I think that pki-ca is going to need a Conflicts ipa < 3.0 in it. certificate renewal failed. I spent far too long trying to figure out why tomcat wasn't listening on port 9180 but failed. I think 9180 is actually the old server, right? So another missing dependency on a fixed certmonger? The best I could find was the certmonger error: ca-error: Error 7 connecting to http://edsel.example.com:9180/ca/ee/ca/profileSubmit: Couldn't connect to server. There is no man page for pkispawn/pkidestroy :-( According to the FHS these should not be in /bin but in /usr/sbin (not end-user commands). The output of pkicreate/pkisilent was really terrible and not usable at all so we didn't display it when failures occurred. It looks like that has been addressed, at least for the case where a CA is already configured and you try to install IPA. Perhaps we should capture stderr and display that instead of the command-line of pkispawn? Again, a man page would help with the integration. 2012-09-10T20:51:45Z DEBUG [2/18]: configuring certificate server instance 2012-09-10T20:51:45Z DEBUG args=/bin/pkispawn -s CA -f /tmp/tmp_Urraq 2012-09-10T20:51:45Z DEBUG stdout= 2012-09-10T20:51:45Z DEBUG stderr=pkispawn : ERROR ....... PKI subsystem 'CA' for instance 'pki-tomcat' already exists! 2012-09-10T20:51:45Z CRITICAL failed to configure ca instance Command '/bin/pkispawn -s CA -f /tmp/tmp_Urraq' returned non-zero exit status 1 rob From rcritten at redhat.com Mon Sep 10 21:38:47 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 10 Sep 2012 17:38:47 -0400 Subject: [Freeipa-devel] [PATCH] 1054 new default SELinux user map user Message-ID: <504E5DE7.804@redhat.com> We've decided to change the default SELinux user map user to the OS default which is unconfined_u. It would be too drastic to go from one extreme to another. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-rcrit-1054-selinux.patch Type: text/x-diff Size: 1684 bytes Desc: not available URL: From jhrozek at redhat.com Mon Sep 10 21:54:02 2012 From: jhrozek at redhat.com (Jakub Hrozek) Date: Mon, 10 Sep 2012 23:54:02 +0200 Subject: [Freeipa-devel] [PATCH] 1054 new default SELinux user map user In-Reply-To: <504E5DE7.804@redhat.com> References: <504E5DE7.804@redhat.com> Message-ID: <20120910215402.GE11130@hendrix.redhat.com> On Mon, Sep 10, 2012 at 05:38:47PM -0400, Rob Crittenden wrote: > We've decided to change the default SELinux user map user to the OS > default which is unconfined_u. It would be too drastic to go from > one extreme to another. > > rob How does one set an "empty default" that the SSSD would treat as "don't create any login file whatsoever" ? From nalin at redhat.com Mon Sep 10 22:24:57 2012 From: nalin at redhat.com (Nalin Dahyabhai) Date: Mon, 10 Sep 2012 18:24:57 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <504E5480.3000201@redhat.com> References: <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> Message-ID: <20120910222457.GE8154@redhat.com> On Mon, Sep 10, 2012 at 04:58:40PM -0400, Rob Crittenden wrote: > certificate renewal failed. I spent far too long trying to figure > out why tomcat wasn't listening on port 9180 but failed. I think > 9180 is actually the old server, right? So another missing > dependency on a fixed certmonger? > > The best I could find was the certmonger error: > > ca-error: Error 7 connecting to > http://edsel.example.com:9180/ca/ee/ca/profileSubmit: Couldn't > connect to server. That's the old port, alright, though I thought the upgrade process wasn't going to be converting already-installed Dogtag instances. Anyway, if your IPA configuration says "dogtag_version = 10", certmonger's not going to notice it until version 0.60. I've just set the wheels in motion to push that version to Fedora 17 and later. HTH, Nalin From alee at redhat.com Tue Sep 11 02:04:25 2012 From: alee at redhat.com (Ade Lee) Date: Mon, 10 Sep 2012 22:04:25 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <504E5480.3000201@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502B86F5.4070008@redhat.com> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> Message-ID: <1347329065.24731.101.camel@aleeredhat.laptop> On Mon, 2012-09-10 at 16:58 -0400, Rob Crittenden wrote: > Petr Viktorin wrote: > > Attaching rebased and squashed patches. I've done some testing with them > > but please test some more. > > > > Most of these aren't IPA issues, but dogtag issues. I'll try to split > them out. > > IPA: > > For the configuration files in install/conf to be updated at rpm update > time the VERSION needs to be incremented. > > The ipa package lacks any updated dogtag dependencies, so I abused it. > > I installed IPA with dogtag 9 and created a replica. > > I updated the IPA bits, that worked fine. > > I updated to dogtag 10 and now the CA doesn't work on the master, > including starting the dogtag instance. Note that the rpm update process > worked, no notice that the CA service didn't restart. > Did you try to restart the CA with selinux in permissive mode? This is still required right now until I get the selinux policy all straightened out. There is also a separate dogtag ticket (which is currently being worked on) to restart existing dogtag instances when dogtag is upgraded from 9->10. > Uninstalling failed because it tried to run pkidestroy and not pkiremove. > I'll test this too. > The contents of the file passed to pkispawn should be logged so we can > see exactly what was passed in. > Its a pretty big file. You might want to only log the modifications. Or save the file somewhere. > DOGTAG: > > When upgrading using the dogtag-devel repo I had to specify > pki-tools.x86_64 otherwise it tried to install both 32 and 64-bit > versions (and failed). > > I ended up running: yum update pki-ca tomcatjss pki-tools.x86_64 > --enablerepo=dogtag-devel --enablerepo=updates-testing > We'll look into this. I think I know why this happens. > What happens if someone manually upgrades pki-ca without first updating > ipa? I think that pki-ca is going to need a Conflicts ipa < 3.0 in it. We can add that. > certificate renewal failed. I spent far too long trying to figure out > why tomcat wasn't listening on port 9180 but failed. I think 9180 is > actually the old server, right? So another missing dependency on a fixed > certmonger? > > The best I could find was the certmonger error: > > ca-error: Error 7 connecting to > http://edsel.example.com:9180/ca/ee/ca/profileSubmit: Couldn't connect > to server. > Is this cert renewal on a dogtag 10 instance? Or the upgraded dogtag 9? If its dogtag 10, perhaps you do not have the certmonger version that has the relevant fix? If its dogtag 9, then we need to figure out whats going on. That reminds me - I need to file a bug to allow certmonger to talk to the newly defined dogtag ports. Do you have selinux permissive? > There is no man page for pkispawn/pkidestroy :-( According to the FHS > these should not be in /bin but in /usr/sbin (not end-user commands). > There is a trac ticket open for man pages for pkispawn and pkidestroy. We plan to complete this ticket by the time f18 is released. We'll look into the location of pkispawn/pkicreate. > The output of pkicreate/pkisilent was really terrible and not usable at > all so we didn't display it when failures occurred. It looks like that > has been addressed, at least for the case where a CA is already > configured and you try to install IPA. Perhaps we should capture stderr > and display that instead of the command-line of pkispawn? Again, a man > page would help with the integration. > > 2012-09-10T20:51:45Z DEBUG [2/18]: configuring certificate server instance > 2012-09-10T20:51:45Z DEBUG args=/bin/pkispawn -s CA -f /tmp/tmp_Urraq > 2012-09-10T20:51:45Z DEBUG stdout= > 2012-09-10T20:51:45Z DEBUG stderr=pkispawn : ERROR ....... PKI > subsystem 'CA' for instance 'pki-tomcat' already exists! > > 2012-09-10T20:51:45Z CRITICAL failed to configure ca instance Command > '/bin/pkispawn -s CA -f /tmp/tmp_Urraq' returned non-zero exit status 1 > That may be a good idea. Of course. thats an IPA thing, right? > rob > From mkosek at redhat.com Tue Sep 11 11:14:39 2012 From: mkosek at redhat.com (Martin Kosek) Date: Tue, 11 Sep 2012 13:14:39 +0200 Subject: [Freeipa-devel] [PATCH 0011] Make sure selinuxusemap behaves consistently to HBAC rule In-Reply-To: <50488561.7090902@redhat.com> References: <5044C8CE.20108@redhat.com> <50473DDF.8040303@redhat.com> <50488561.7090902@redhat.com> Message-ID: <504F1D1F.8070505@redhat.com> On 09/06/2012 01:13 PM, Tomas Babej wrote: > On 09/05/2012 01:56 PM, Martin Kosek wrote: >> On 09/03/2012 05:12 PM, Tomas Babej wrote: >>> Hi, >>> >>> Both selinuxusermap-add and selinuxusermap-mod commands now behave >>> consistently in not allowing user/host category or user/host members >>> and HBAC rule being set at the same time. Also adds a bunch of unit >>> tests that check this behaviour. >>> >>> https://fedorahosted.org/freeipa/ticket/2983 >>> >>> Tomas >>> >> I found few issues with this patch: >> >> 1) Patch needs a rebase >> >> 2) Patch does not expect attributes to be set to None, i.e. to be left empty or >> to be deleted, e.g.: >> >> # ipa selinuxusermap-add foo --selinuxuser=guest_u:s0 --usercat=all --hbacrule= >> ipa: ERROR: HBAC rule and local members cannot both be set >> >> # ipa selinuxusermap-add foo --selinuxuser=guest_u:s0 --usercat=all >> ---------------------------- >> Added SELinux User Map "foo" >> ---------------------------- >> Rule name: foo >> SELinux User: guest_u:s0 >> User category: all >> Enabled: TRUE >> >> # ipa selinuxusermap-mod foo --usercat= --hbacrule= >> ipa: ERROR: HBAC rule and local members cannot both be set >> >> # ipa selinuxusermap-mod foo --usercat= >> ------------------------------- >> Modified SELinux User Map "foo" >> ------------------------------- >> Rule name: foo >> SELinux User: guest_u:s0 >> Enabled: TRUE >> >> # ipa selinuxusermap-mod foo --hbacrule=foo >> ------------------------------- >> Modified SELinux User Map "foo" >> ------------------------------- >> Rule name: foo >> SELinux User: guest_u:s0 >> HBAC Rule: foo >> Enabled: TRUE >> >> # ipa selinuxusermap-mod foo --hbacrule= --usercat=all >> ipa: ERROR: HBAC rule and local members cannot both be set >> >> All these validation failures are not valid. >> >> 3) Additionally, I think it would be more readable and less error prone that if >> instead of this blob: >> >> + are_local_members_to_be_set = 'usercategory' in _entry_attrs or \ >> + 'hostcategory' in _entry_attrs or \ >> + 'memberuser' in _entry_attrs or \ >> + 'memberhost' in _entry_attrs >> >> You would use something like that: >> >> are_local_members_to_be_set = any(attr in _entry_attrs >> for attr in ('usercategory', >> 'hostcategory', >> 'memberuser', >> 'memberhost')) >> >> Martin > 1.) Done. > 2.) Corrected. > 3.) Fixed. > > Tomas 1) There are some (corner) cases where this approach still does not work: # ipa selinuxusermap-show foo Rule name: foo SELinux User: guest_u:s0 HBAC Rule: foo Enabled: TRUE # ipa selinuxusermap-mod foo --usercat=all --hbacrule= ipa: ERROR: HBAC rule and local members cannot both be set HBAC rule attribute is being deleted and user category set, so this should not be rejected. 2) There are also some styling issues (you can use pep8 tool present in Fedora to locate them on your own, e.g.: ipalib/plugins/selinuxusermap.py:247:32: E203 whitespace before ':' ipalib/plugins/selinuxusermap.py:247:70: E225 missing whitespace around operator ipalib/plugins/selinuxusermap.py:249:36: E221 multiple spaces before operator ... Martin From pviktori at redhat.com Tue Sep 11 12:41:51 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Tue, 11 Sep 2012 14:41:51 +0200 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <1347329065.24731.101.camel@aleeredhat.laptop> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502B86F5.4070008@redhat.com> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> <1347329065.24731.101.camel@aleeredhat.laptop> Message-ID: <504F318F.3030603@redhat.com> On 09/11/2012 04:04 AM, Ade Lee wrote: > On Mon, 2012-09-10 at 16:58 -0400, Rob Crittenden wrote: >> Petr Viktorin wrote: >>> Attaching rebased and squashed patches. I've done some testing with them >>> but please test some more. >>> >> >> Most of these aren't IPA issues, but dogtag issues. I'll try to split >> them out. >> >> IPA: >> >> For the configuration files in install/conf to be updated at rpm update >> time the VERSION needs to be incremented. These files should stay the same since on upgrade we're still using a Dogtag 9 style instance. The Dogtag 10 ports are only used in new installs. >> The ipa package lacks any updated dogtag dependencies, so I abused it. What should the updated dependencies be? Since it should work with both dogtag 9 and 10, I don't see how they should change. >> I installed IPA with dogtag 9 and created a replica. >> >> I updated the IPA bits, that worked fine. >> >> I updated to dogtag 10 and now the CA doesn't work on the master, >> including starting the dogtag instance. Note that the rpm update process >> worked, no notice that the CA service didn't restart. >> > Did you try to restart the CA with selinux in permissive mode? This is > still required right now until I get the selinux policy all straightened > out. > > There is also a separate dogtag ticket (which is currently being worked > on) to restart existing dogtag instances when dogtag is upgraded from > 9->10. In permissive mode, this upgrade works for me. Sometimes I do get this error intermittently: ipa: ERROR: Certificate operation cannot be completed: Unable to communicate with CMS (Service Temporarily Unavailable) Usually, waiting a couple of minutes clears this up. Perhaps we are not doing startup detection correctly. Ade mentioned that waiting for ports may not be ideal. How do we know if Dogtag is initialized? >> Uninstalling failed because it tried to run pkidestroy and not pkiremove. I was under the impression that pkidestroy was the correct command to remove an upgraded instance. I'll check with Ade. > I'll test this too. > >> The contents of the file passed to pkispawn should be logged so we can >> see exactly what was passed in. >> > Its a pretty big file. You might want to only log the modifications. > Or save the file somewhere. Our logs are pretty verbose, so that shouldn't be a problem. I'll put it in the next version of the patch. >> DOGTAG: >> >> When upgrading using the dogtag-devel repo I had to specify >> pki-tools.x86_64 otherwise it tried to install both 32 and 64-bit >> versions (and failed). >> >> I ended up running: yum update pki-ca tomcatjss pki-tools.x86_64 >> --enablerepo=dogtag-devel --enablerepo=updates-testing >> > We'll look into this. I think I know why this happens. > >> What happens if someone manually upgrades pki-ca without first updating >> ipa? I think that pki-ca is going to need a Conflicts ipa < 3.0 in it. > > We can add that. > >> certificate renewal failed. I spent far too long trying to figure out >> why tomcat wasn't listening on port 9180 but failed. I think 9180 is >> actually the old server, right? So another missing dependency on a fixed >> certmonger? >> >> The best I could find was the certmonger error: >> >> ca-error: Error 7 connecting to >> http://edsel.example.com:9180/ca/ee/ca/profileSubmit: Couldn't connect >> to server. >> I'll set the certmonger min version to 0.60 as per Nalin's mail. > Is this cert renewal on a dogtag 10 instance? Or the upgraded dogtag 9? > If its dogtag 10, perhaps you do not have the certmonger version that > has the relevant fix? If its dogtag 9, then we need to figure out whats > going on. That reminds me - I need to file a bug to allow certmonger to > talk to the newly defined dogtag ports. Do you have selinux permissive? > >> There is no man page for pkispawn/pkidestroy :-( According to the FHS >> these should not be in /bin but in /usr/sbin (not end-user commands). >> > There is a trac ticket open for man pages for pkispawn and pkidestroy. > We plan to complete this ticket by the time f18 is released. > > We'll look into the location of pkispawn/pkicreate. > >> The output of pkicreate/pkisilent was really terrible and not usable at >> all so we didn't display it when failures occurred. It looks like that >> has been addressed, at least for the case where a CA is already >> configured and you try to install IPA. Perhaps we should capture stderr >> and display that instead of the command-line of pkispawn? Again, a man >> page would help with the integration. >> >> 2012-09-10T20:51:45Z DEBUG [2/18]: configuring certificate server instance >> 2012-09-10T20:51:45Z DEBUG args=/bin/pkispawn -s CA -f /tmp/tmp_Urraq >> 2012-09-10T20:51:45Z DEBUG stdout= >> 2012-09-10T20:51:45Z DEBUG stderr=pkispawn : ERROR ....... PKI >> subsystem 'CA' for instance 'pki-tomcat' already exists! >> >> 2012-09-10T20:51:45Z CRITICAL failed to configure ca instance Command >> '/bin/pkispawn -s CA -f /tmp/tmp_Urraq' returned non-zero exit status 1 >> > That may be a good idea. Of course. thats an IPA thing, right? > >> rob >> > > -- Petr? From rcritten at redhat.com Tue Sep 11 12:59:25 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 11 Sep 2012 08:59:25 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <504F318F.3030603@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502B86F5.4070008@redhat.com> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> <1347329065.24731.101.camel@aleeredhat.laptop> <504F318F.3030603@redhat.com> Message-ID: <504F35AD.3030905@redhat.com> Petr Viktorin wrote: > On 09/11/2012 04:04 AM, Ade Lee wrote: >> On Mon, 2012-09-10 at 16:58 -0400, Rob Crittenden wrote: >>> Petr Viktorin wrote: >>>> Attaching rebased and squashed patches. I've done some testing with >>>> them >>>> but please test some more. >>>> >>> >>> Most of these aren't IPA issues, but dogtag issues. I'll try to split >>> them out. >>> >>> IPA: >>> >>> For the configuration files in install/conf to be updated at rpm update >>> time the VERSION needs to be incremented. > > These files should stay the same since on upgrade we're still using a > Dogtag 9 style instance. The Dogtag 10 ports are only used in new installs. > >>> The ipa package lacks any updated dogtag dependencies, so I abused it. > > What should the updated dependencies be? Since it should work with both > dogtag 9 and 10, I don't see how they should change. I don't know either, but we need to prevent people from installing incompatible package combinations. >>> I installed IPA with dogtag 9 and created a replica. >>> >>> I updated the IPA bits, that worked fine. >>> >>> I updated to dogtag 10 and now the CA doesn't work on the master, >>> including starting the dogtag instance. Note that the rpm update process >>> worked, no notice that the CA service didn't restart. >>> >> Did you try to restart the CA with selinux in permissive mode? This is >> still required right now until I get the selinux policy all straightened >> out. >> >> There is also a separate dogtag ticket (which is currently being worked >> on) to restart existing dogtag instances when dogtag is upgraded from >> 9->10. > > In permissive mode, this upgrade works for me. I was in enforcing mode but saw no AVCs. What is the ETA on fixing this? > > > Sometimes I do get this error intermittently: > > ipa: ERROR: Certificate operation cannot be completed: Unable to > communicate with CMS (Service Temporarily Unavailable) > > Usually, waiting a couple of minutes clears this up. Perhaps we are not > doing startup detection correctly. Ade mentioned that waiting for ports > may not be ideal. How do we know if Dogtag is initialized? Years ago we had discussed with Andrew and Matt creating a URI that can be queried to determine dogtag status. I don't think that ever went anywhere. > >>> Uninstalling failed because it tried to run pkidestroy and not >>> pkiremove. > > I was under the impression that pkidestroy was the correct command to > remove an upgraded instance. I'll check with Ade. > >> I'll test this too. >> >>> The contents of the file passed to pkispawn should be logged so we can >>> see exactly what was passed in. >>> >> Its a pretty big file. You might want to only log the modifications. >> Or save the file somewhere. > > Our logs are pretty verbose, so that shouldn't be a problem. I'll put it > in the next version of the patch. The question to ask is: would you need the contents of this file if all you got were logs and needed to evaluate why installation failed? In most cases this is yes. >>> DOGTAG: >>> >>> When upgrading using the dogtag-devel repo I had to specify >>> pki-tools.x86_64 otherwise it tried to install both 32 and 64-bit >>> versions (and failed). >>> >>> I ended up running: yum update pki-ca tomcatjss pki-tools.x86_64 >>> --enablerepo=dogtag-devel --enablerepo=updates-testing >>> >> We'll look into this. I think I know why this happens. >> >>> What happens if someone manually upgrades pki-ca without first updating >>> ipa? I think that pki-ca is going to need a Conflicts ipa < 3.0 in it. >> >> We can add that. >> >>> certificate renewal failed. I spent far too long trying to figure out >>> why tomcat wasn't listening on port 9180 but failed. I think 9180 is >>> actually the old server, right? So another missing dependency on a fixed >>> certmonger? >>> >>> The best I could find was the certmonger error: >>> >>> ca-error: Error 7 connecting to >>> http://edsel.example.com:9180/ca/ee/ca/profileSubmit: Couldn't connect >>> to server. >>> > > I'll set the certmonger min version to 0.60 as per Nalin's mail. Ok. >> Is this cert renewal on a dogtag 10 instance? Or the upgraded dogtag 9? >> If its dogtag 10, perhaps you do not have the certmonger version that >> has the relevant fix? If its dogtag 9, then we need to figure out whats >> going on. That reminds me - I need to file a bug to allow certmonger to >> talk to the newly defined dogtag ports. Do you have selinux permissive? >> >>> There is no man page for pkispawn/pkidestroy :-( According to the FHS >>> these should not be in /bin but in /usr/sbin (not end-user commands). >>> >> There is a trac ticket open for man pages for pkispawn and pkidestroy. >> We plan to complete this ticket by the time f18 is released. >> >> We'll look into the location of pkispawn/pkicreate. >> >>> The output of pkicreate/pkisilent was really terrible and not usable at >>> all so we didn't display it when failures occurred. It looks like that >>> has been addressed, at least for the case where a CA is already >>> configured and you try to install IPA. Perhaps we should capture stderr >>> and display that instead of the command-line of pkispawn? Again, a man >>> page would help with the integration. >>> >>> 2012-09-10T20:51:45Z DEBUG [2/18]: configuring certificate server >>> instance >>> 2012-09-10T20:51:45Z DEBUG args=/bin/pkispawn -s CA -f /tmp/tmp_Urraq >>> 2012-09-10T20:51:45Z DEBUG stdout= >>> 2012-09-10T20:51:45Z DEBUG stderr=pkispawn : ERROR ....... PKI >>> subsystem 'CA' for instance 'pki-tomcat' already exists! >>> >>> 2012-09-10T20:51:45Z CRITICAL failed to configure ca instance Command >>> '/bin/pkispawn -s CA -f /tmp/tmp_Urraq' returned non-zero exit status 1 >>> >> That may be a good idea. Of course. thats an IPA thing, right? Right, not a show-stopper for this but a new ticket should be opened to track it. rob From pviktori at redhat.com Tue Sep 11 14:18:32 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Tue, 11 Sep 2012 16:18:32 +0200 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <504F35AD.3030905@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502B86F5.4070008@redhat.com> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> <1347329065.24731.101.camel@aleeredhat.laptop> <504F318F.3030603@redhat.com> <504F35AD.3030905@redhat.com> Message-ID: <504F4838.4040009@redhat.com> I clarified with Ade that d9-style instances need pkiremove even under d10, and went through other items that changed to verify what they do in this case. I put the set of constants into namespace classes to simplify handling them, since more parts of CAInstance need bits for the configured version that Ade's patch suggested. I've added the pkispawn config file to the logs, and bumped the certmonger version in the specfile. On 09/11/2012 02:59 PM, Rob Crittenden wrote: > Petr Viktorin wrote: >> On 09/11/2012 04:04 AM, Ade Lee wrote: >>> On Mon, 2012-09-10 at 16:58 -0400, Rob Crittenden wrote: >>>> Petr Viktorin wrote: >>>>> Attaching rebased and squashed patches. I've done some testing with >>>>> them >>>>> but please test some more. >>>>> >>>> >>>> Most of these aren't IPA issues, but dogtag issues. I'll try to split >>>> them out. >>>> >>>> IPA: >>>> >>>> For the configuration files in install/conf to be updated at rpm update >>>> time the VERSION needs to be incremented. >> >> These files should stay the same since on upgrade we're still using a >> Dogtag 9 style instance. The Dogtag 10 ports are only used in new >> installs. >> >>>> The ipa package lacks any updated dogtag dependencies, so I abused it. >> >> What should the updated dependencies be? Since it should work with both >> dogtag 9 and 10, I don't see how they should change. > > I don't know either, but we need to prevent people from installing > incompatible package combinations. > >>>> I installed IPA with dogtag 9 and created a replica. >>>> >>>> I updated the IPA bits, that worked fine. >>>> >>>> I updated to dogtag 10 and now the CA doesn't work on the master, >>>> including starting the dogtag instance. Note that the rpm update >>>> process >>>> worked, no notice that the CA service didn't restart. >>>> >>> Did you try to restart the CA with selinux in permissive mode? This is >>> still required right now until I get the selinux policy all straightened >>> out. >>> >>> There is also a separate dogtag ticket (which is currently being worked >>> on) to restart existing dogtag instances when dogtag is upgraded from >>> 9->10. >> >> In permissive mode, this upgrade works for me. > > I was in enforcing mode but saw no AVCs. What is the ETA on fixing this? > >> >> >> Sometimes I do get this error intermittently: >> >> ipa: ERROR: Certificate operation cannot be completed: Unable to >> communicate with CMS (Service Temporarily Unavailable) >> >> Usually, waiting a couple of minutes clears this up. Perhaps we are not >> doing startup detection correctly. Ade mentioned that waiting for ports >> may not be ideal. How do we know if Dogtag is initialized? > > Years ago we had discussed with Andrew and Matt creating a URI that can > be queried to determine dogtag status. I don't think that ever went > anywhere. > >> >>>> Uninstalling failed because it tried to run pkidestroy and not >>>> pkiremove. >> >> I was under the impression that pkidestroy was the correct command to >> remove an upgraded instance. I'll check with Ade. >> >>> I'll test this too. >>> >>>> The contents of the file passed to pkispawn should be logged so we can >>>> see exactly what was passed in. >>>> >>> Its a pretty big file. You might want to only log the modifications. >>> Or save the file somewhere. >> >> Our logs are pretty verbose, so that shouldn't be a problem. I'll put it >> in the next version of the patch. > > The question to ask is: would you need the contents of this file if all > you got were logs and needed to evaluate why installation failed? In > most cases this is yes. > >>>> DOGTAG: >>>> >>>> When upgrading using the dogtag-devel repo I had to specify >>>> pki-tools.x86_64 otherwise it tried to install both 32 and 64-bit >>>> versions (and failed). >>>> >>>> I ended up running: yum update pki-ca tomcatjss pki-tools.x86_64 >>>> --enablerepo=dogtag-devel --enablerepo=updates-testing >>>> >>> We'll look into this. I think I know why this happens. >>> >>>> What happens if someone manually upgrades pki-ca without first updating >>>> ipa? I think that pki-ca is going to need a Conflicts ipa < 3.0 in it. >>> >>> We can add that. >>> >>>> certificate renewal failed. I spent far too long trying to figure out >>>> why tomcat wasn't listening on port 9180 but failed. I think 9180 is >>>> actually the old server, right? So another missing dependency on a >>>> fixed >>>> certmonger? >>>> >>>> The best I could find was the certmonger error: >>>> >>>> ca-error: Error 7 connecting to >>>> http://edsel.example.com:9180/ca/ee/ca/profileSubmit: Couldn't connect >>>> to server. >>>> >> >> I'll set the certmonger min version to 0.60 as per Nalin's mail. > > Ok. > >>> Is this cert renewal on a dogtag 10 instance? Or the upgraded dogtag 9? >>> If its dogtag 10, perhaps you do not have the certmonger version that >>> has the relevant fix? If its dogtag 9, then we need to figure out whats >>> going on. That reminds me - I need to file a bug to allow certmonger to >>> talk to the newly defined dogtag ports. Do you have selinux permissive? >>> >>>> There is no man page for pkispawn/pkidestroy :-( According to the FHS >>>> these should not be in /bin but in /usr/sbin (not end-user commands). >>>> >>> There is a trac ticket open for man pages for pkispawn and pkidestroy. >>> We plan to complete this ticket by the time f18 is released. >>> >>> We'll look into the location of pkispawn/pkicreate. >>> >>>> The output of pkicreate/pkisilent was really terrible and not usable at >>>> all so we didn't display it when failures occurred. It looks like that >>>> has been addressed, at least for the case where a CA is already >>>> configured and you try to install IPA. Perhaps we should capture stderr >>>> and display that instead of the command-line of pkispawn? Again, a man >>>> page would help with the integration. >>>> >>>> 2012-09-10T20:51:45Z DEBUG [2/18]: configuring certificate server >>>> instance >>>> 2012-09-10T20:51:45Z DEBUG args=/bin/pkispawn -s CA -f /tmp/tmp_Urraq >>>> 2012-09-10T20:51:45Z DEBUG stdout= >>>> 2012-09-10T20:51:45Z DEBUG stderr=pkispawn : ERROR ....... PKI >>>> subsystem 'CA' for instance 'pki-tomcat' already exists! >>>> >>>> 2012-09-10T20:51:45Z CRITICAL failed to configure ca instance Command >>>> '/bin/pkispawn -s CA -f /tmp/tmp_Urraq' returned non-zero exit status 1 >>>> >>> That may be a good idea. Of course. thats an IPA thing, right? > > Right, not a show-stopper for this but a new ticket should be opened to > track it. > > rob > -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0074-05-Use-Dogtag-10-only-when-it-is-available.patch Type: text/x-patch Size: 57887 bytes Desc: not available URL: From alee at redhat.com Tue Sep 11 14:28:11 2012 From: alee at redhat.com (Ade Lee) Date: Tue, 11 Sep 2012 10:28:11 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <504F35AD.3030905@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502B86F5.4070008@redhat.com> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> <1347329065.24731.101.camel@aleeredhat.laptop> <504F318F.3030603@redhat.com> <504F35AD.3030905@redhat.com> Message-ID: <1347373691.24731.125.camel@aleeredhat.laptop> On Tue, 2012-09-11 at 08:59 -0400, Rob Crittenden wrote: > Petr Viktorin wrote: > > On 09/11/2012 04:04 AM, Ade Lee wrote: > >> On Mon, 2012-09-10 at 16:58 -0400, Rob Crittenden wrote: > >>> Petr Viktorin wrote: > >>>> Attaching rebased and squashed patches. I've done some testing with > >>>> them > >>>> but please test some more. > >>>> > >>> > >>> Most of these aren't IPA issues, but dogtag issues. I'll try to split > >>> them out. > >>> > >>> IPA: > >>> > >>> For the configuration files in install/conf to be updated at rpm update > >>> time the VERSION needs to be incremented. > > > > These files should stay the same since on upgrade we're still using a > > Dogtag 9 style instance. The Dogtag 10 ports are only used in new installs. > > > >>> The ipa package lacks any updated dogtag dependencies, so I abused it. > > > > What should the updated dependencies be? Since it should work with both > > dogtag 9 and 10, I don't see how they should change. > > I don't know either, but we need to prevent people from installing > incompatible package combinations. > Would'nt the Conflicts: ipa < 3.0 in pki-ca mentioned below satisfy this requirement? The main concern is that you must have ipa 3.0 if you have dogtag 10. Given that dogtag is consumed by IPA though, it makes more sense to put the relevant conflicts in IPA rather than in dogtag. So in this case, that would mean putting Conflicts: pki-ca >= 10.0 in IPA 2.x. Recall that dogtag 10 will only be officially available in f18+. > >>> I installed IPA with dogtag 9 and created a replica. > >>> > >>> I updated the IPA bits, that worked fine. > >>> > >>> I updated to dogtag 10 and now the CA doesn't work on the master, > >>> including starting the dogtag instance. Note that the rpm update process > >>> worked, no notice that the CA service didn't restart. > >>> > >> Did you try to restart the CA with selinux in permissive mode? This is > >> still required right now until I get the selinux policy all straightened > >> out. > >> > >> There is also a separate dogtag ticket (which is currently being worked > >> on) to restart existing dogtag instances when dogtag is upgraded from > >> 9->10. > > > > In permissive mode, this upgrade works for me. > > I was in enforcing mode but saw no AVCs. What is the ETA on fixing this? > Within the next week or two, I need to finish the IPA merge database patch first, and then co-ordinate changes with the selinux guys. > > > > > > Sometimes I do get this error intermittently: > > > > ipa: ERROR: Certificate operation cannot be completed: Unable to > > communicate with CMS (Service Temporarily Unavailable) > > > > Usually, waiting a couple of minutes clears this up. Perhaps we are not > > doing startup detection correctly. Ade mentioned that waiting for ports > > may not be ideal. How do we know if Dogtag is initialized? > > Years ago we had discussed with Andrew and Matt creating a URI that can > be queried to determine dogtag status. I don't think that ever went > anywhere. > Petr, this happens only on reboot, right? And not on regular "service ipa restart"? Yeah, I remember this conversation - and even created a bug for it at some point. This went away because the mechanism you were using seemed to be working. The timing may be very different now with tomcat 7 and systemd. I'll open a dogtag trac ticket for this. > > > >>> Uninstalling failed because it tried to run pkidestroy and not > >>> pkiremove. > > > > I was under the impression that pkidestroy was the correct command to > > remove an upgraded instance. I'll check with Ade. > > > >> I'll test this too. > >> > >>> The contents of the file passed to pkispawn should be logged so we can > >>> see exactly what was passed in. > >>> > >> Its a pretty big file. You might want to only log the modifications. > >> Or save the file somewhere. > > > > Our logs are pretty verbose, so that shouldn't be a problem. I'll put it > > in the next version of the patch. > > The question to ask is: would you need the contents of this file if all > you got were logs and needed to evaluate why installation failed? In > most cases this is yes. > Up to you guys. There is a patch I am working on in which I will be logging the object that is being passed to the server from pkispawn. That - and the diffs to the standard config file as I mentioned above - will likely be sufficient to debug almost all cases. Make sure not to log any passwords. > >>> DOGTAG: > >>> > >>> When upgrading using the dogtag-devel repo I had to specify > >>> pki-tools.x86_64 otherwise it tried to install both 32 and 64-bit > >>> versions (and failed). > >>> > >>> I ended up running: yum update pki-ca tomcatjss pki-tools.x86_64 > >>> --enablerepo=dogtag-devel --enablerepo=updates-testing > >>> > >> We'll look into this. I think I know why this happens. > >> > >>> What happens if someone manually upgrades pki-ca without first updating > >>> ipa? I think that pki-ca is going to need a Conflicts ipa < 3.0 in it. > >> > >> We can add that. > >> > >>> certificate renewal failed. I spent far too long trying to figure out > >>> why tomcat wasn't listening on port 9180 but failed. I think 9180 is > >>> actually the old server, right? So another missing dependency on a fixed > >>> certmonger? > >>> > >>> The best I could find was the certmonger error: > >>> > >>> ca-error: Error 7 connecting to > >>> http://edsel.example.com:9180/ca/ee/ca/profileSubmit: Couldn't connect > >>> to server. > >>> > > > > I'll set the certmonger min version to 0.60 as per Nalin's mail. > > Ok. > > >> Is this cert renewal on a dogtag 10 instance? Or the upgraded dogtag 9? > >> If its dogtag 10, perhaps you do not have the certmonger version that > >> has the relevant fix? If its dogtag 9, then we need to figure out whats > >> going on. That reminds me - I need to file a bug to allow certmonger to > >> talk to the newly defined dogtag ports. Do you have selinux permissive? > >> > >>> There is no man page for pkispawn/pkidestroy :-( According to the FHS > >>> these should not be in /bin but in /usr/sbin (not end-user commands). > >>> > >> There is a trac ticket open for man pages for pkispawn and pkidestroy. > >> We plan to complete this ticket by the time f18 is released. > >> > >> We'll look into the location of pkispawn/pkicreate. > >> > >>> The output of pkicreate/pkisilent was really terrible and not usable at > >>> all so we didn't display it when failures occurred. It looks like that > >>> has been addressed, at least for the case where a CA is already > >>> configured and you try to install IPA. Perhaps we should capture stderr > >>> and display that instead of the command-line of pkispawn? Again, a man > >>> page would help with the integration. > >>> > >>> 2012-09-10T20:51:45Z DEBUG [2/18]: configuring certificate server > >>> instance > >>> 2012-09-10T20:51:45Z DEBUG args=/bin/pkispawn -s CA -f /tmp/tmp_Urraq > >>> 2012-09-10T20:51:45Z DEBUG stdout= > >>> 2012-09-10T20:51:45Z DEBUG stderr=pkispawn : ERROR ....... PKI > >>> subsystem 'CA' for instance 'pki-tomcat' already exists! > >>> > >>> 2012-09-10T20:51:45Z CRITICAL failed to configure ca instance Command > >>> '/bin/pkispawn -s CA -f /tmp/tmp_Urraq' returned non-zero exit status 1 > >>> > >> That may be a good idea. Of course. thats an IPA thing, right? > > Right, not a show-stopper for this but a new ticket should be opened to > track it. > > rob > From rcritten at redhat.com Tue Sep 11 14:38:24 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 11 Sep 2012 10:38:24 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <1347373691.24731.125.camel@aleeredhat.laptop> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502B86F5.4070008@redhat.com> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> <1347329065.24731.101.camel@aleeredhat.laptop> <504F318F.3030603@redhat.com> <504F35AD.3030905@redhat.com> <1347373691.24731.125.camel@aleeredhat.laptop> Message-ID: <504F4CE0.9080706@redhat.com> Ade Lee wrote: > On Tue, 2012-09-11 at 08:59 -0400, Rob Crittenden wrote: >> Petr Viktorin wrote: >>> On 09/11/2012 04:04 AM, Ade Lee wrote: >>>> On Mon, 2012-09-10 at 16:58 -0400, Rob Crittenden wrote: >>>>> Petr Viktorin wrote: >>>>>> Attaching rebased and squashed patches. I've done some testing with >>>>>> them >>>>>> but please test some more. >>>>>> >>>>> >>>>> Most of these aren't IPA issues, but dogtag issues. I'll try to split >>>>> them out. >>>>> >>>>> IPA: >>>>> >>>>> For the configuration files in install/conf to be updated at rpm update >>>>> time the VERSION needs to be incremented. >>> >>> These files should stay the same since on upgrade we're still using a >>> Dogtag 9 style instance. The Dogtag 10 ports are only used in new installs. >>> >>>>> The ipa package lacks any updated dogtag dependencies, so I abused it. >>> >>> What should the updated dependencies be? Since it should work with both >>> dogtag 9 and 10, I don't see how they should change. >> >> I don't know either, but we need to prevent people from installing >> incompatible package combinations. >> > Would'nt the Conflicts: ipa < 3.0 in pki-ca mentioned below satisfy this > requirement? The main concern is that you must have ipa 3.0 if you have > dogtag 10. > > Given that dogtag is consumed by IPA though, it makes more sense to put > the relevant conflicts in IPA rather than in dogtag. So in this case, > that would mean putting Conflicts: pki-ca >= 10.0 in IPA 2.x. > Recall that dogtag 10 will only be officially available in f18+. That isn't enough. If a F-17 user with IPA 2.2 installed upgrades to F-18 they would be able to install dogtag 10 and blow up their IPA server. > >>>>> I installed IPA with dogtag 9 and created a replica. >>>>> >>>>> I updated the IPA bits, that worked fine. >>>>> >>>>> I updated to dogtag 10 and now the CA doesn't work on the master, >>>>> including starting the dogtag instance. Note that the rpm update process >>>>> worked, no notice that the CA service didn't restart. >>>>> >>>> Did you try to restart the CA with selinux in permissive mode? This is >>>> still required right now until I get the selinux policy all straightened >>>> out. >>>> >>>> There is also a separate dogtag ticket (which is currently being worked >>>> on) to restart existing dogtag instances when dogtag is upgraded from >>>> 9->10. >>> >>> In permissive mode, this upgrade works for me. >> >> I was in enforcing mode but saw no AVCs. What is the ETA on fixing this? >> > > Within the next week or two, I need to finish the IPA merge database > patch first, and then co-ordinate changes with the selinux guys. > >>> >>> >>> Sometimes I do get this error intermittently: >>> >>> ipa: ERROR: Certificate operation cannot be completed: Unable to >>> communicate with CMS (Service Temporarily Unavailable) >>> >>> Usually, waiting a couple of minutes clears this up. Perhaps we are not >>> doing startup detection correctly. Ade mentioned that waiting for ports >>> may not be ideal. How do we know if Dogtag is initialized? >> >> Years ago we had discussed with Andrew and Matt creating a URI that can >> be queried to determine dogtag status. I don't think that ever went >> anywhere. >> > Petr, this happens only on reboot, right? And not on regular "service > ipa restart"? > > Yeah, I remember this conversation - and even created a bug for it at > some point. This went away because the mechanism you were using seemed > to be working. The timing may be very different now with tomcat 7 and > systemd. I'll open a dogtag trac ticket for this. Ok. > >>> >>>>> Uninstalling failed because it tried to run pkidestroy and not >>>>> pkiremove. >>> >>> I was under the impression that pkidestroy was the correct command to >>> remove an upgraded instance. I'll check with Ade. >>> >>>> I'll test this too. >>>> >>>>> The contents of the file passed to pkispawn should be logged so we can >>>>> see exactly what was passed in. >>>>> >>>> Its a pretty big file. You might want to only log the modifications. >>>> Or save the file somewhere. >>> >>> Our logs are pretty verbose, so that shouldn't be a problem. I'll put it >>> in the next version of the patch. >> >> The question to ask is: would you need the contents of this file if all >> you got were logs and needed to evaluate why installation failed? In >> most cases this is yes. >> > Up to you guys. There is a patch I am working on in which I will be > logging the object that is being passed to the server from pkispawn. > That - and the diffs to the standard config file as I mentioned above - > will likely be sufficient to debug almost all cases. > > Make sure not to log any passwords. > >>>>> DOGTAG: >>>>> >>>>> When upgrading using the dogtag-devel repo I had to specify >>>>> pki-tools.x86_64 otherwise it tried to install both 32 and 64-bit >>>>> versions (and failed). >>>>> >>>>> I ended up running: yum update pki-ca tomcatjss pki-tools.x86_64 >>>>> --enablerepo=dogtag-devel --enablerepo=updates-testing >>>>> >>>> We'll look into this. I think I know why this happens. >>>> >>>>> What happens if someone manually upgrades pki-ca without first updating >>>>> ipa? I think that pki-ca is going to need a Conflicts ipa < 3.0 in it. >>>> >>>> We can add that. >>>> >>>>> certificate renewal failed. I spent far too long trying to figure out >>>>> why tomcat wasn't listening on port 9180 but failed. I think 9180 is >>>>> actually the old server, right? So another missing dependency on a fixed >>>>> certmonger? >>>>> >>>>> The best I could find was the certmonger error: >>>>> >>>>> ca-error: Error 7 connecting to >>>>> http://edsel.example.com:9180/ca/ee/ca/profileSubmit: Couldn't connect >>>>> to server. >>>>> >>> >>> I'll set the certmonger min version to 0.60 as per Nalin's mail. >> >> Ok. >> >>>> Is this cert renewal on a dogtag 10 instance? Or the upgraded dogtag 9? >>>> If its dogtag 10, perhaps you do not have the certmonger version that >>>> has the relevant fix? If its dogtag 9, then we need to figure out whats >>>> going on. That reminds me - I need to file a bug to allow certmonger to >>>> talk to the newly defined dogtag ports. Do you have selinux permissive? >>>> >>>>> There is no man page for pkispawn/pkidestroy :-( According to the FHS >>>>> these should not be in /bin but in /usr/sbin (not end-user commands). >>>>> >>>> There is a trac ticket open for man pages for pkispawn and pkidestroy. >>>> We plan to complete this ticket by the time f18 is released. >>>> >>>> We'll look into the location of pkispawn/pkicreate. >>>> >>>>> The output of pkicreate/pkisilent was really terrible and not usable at >>>>> all so we didn't display it when failures occurred. It looks like that >>>>> has been addressed, at least for the case where a CA is already >>>>> configured and you try to install IPA. Perhaps we should capture stderr >>>>> and display that instead of the command-line of pkispawn? Again, a man >>>>> page would help with the integration. >>>>> >>>>> 2012-09-10T20:51:45Z DEBUG [2/18]: configuring certificate server >>>>> instance >>>>> 2012-09-10T20:51:45Z DEBUG args=/bin/pkispawn -s CA -f /tmp/tmp_Urraq >>>>> 2012-09-10T20:51:45Z DEBUG stdout= >>>>> 2012-09-10T20:51:45Z DEBUG stderr=pkispawn : ERROR ....... PKI >>>>> subsystem 'CA' for instance 'pki-tomcat' already exists! >>>>> >>>>> 2012-09-10T20:51:45Z CRITICAL failed to configure ca instance Command >>>>> '/bin/pkispawn -s CA -f /tmp/tmp_Urraq' returned non-zero exit status 1 >>>>> >>>> That may be a good idea. Of course. thats an IPA thing, right? >> >> Right, not a show-stopper for this but a new ticket should be opened to >> track it. >> >> rob >> > > From pvoborni at redhat.com Tue Sep 11 15:09:30 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Tue, 11 Sep 2012 17:09:30 +0200 Subject: [Freeipa-devel] [PATCH] 212 Fix integer validation when boundary value is empty string Message-ID: <504F542A.7060204@redhat.com> There was an error in number validation check. If boundary value was an empty string, validation of a number always failed. This patch fixes the problem by not performing the check in these cases. Basic unit tests for IPA.metadata_validator created. -- Petr Vobornik -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pvoborni-0212-Fix-integer-validation-when-boundary-value-is-empty-.patch Type: text/x-patch Size: 9344 bytes Desc: not available URL: From pvoborni at redhat.com Tue Sep 11 15:11:49 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Tue, 11 Sep 2012 17:11:49 +0200 Subject: [Freeipa-devel] [PATCH] 213 JSON serialization of long type Message-ID: <504F54B5.6040901@redhat.com> Numbers of long type were incorrectly serialized to JSON as empty strings when using json_serialize function. It caused problem in serialization of metadata for Web UI. This patch is fixing it. Discovered after "Cast DNS SOA serial maximum boundary to long" -- Petr Vobornik -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pvoborni-0213-JSON-serialization-of-long-type.patch Type: text/x-patch Size: 1150 bytes Desc: not available URL: From pvoborni at redhat.com Tue Sep 11 15:15:32 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Tue, 11 Sep 2012 17:15:32 +0200 Subject: [Freeipa-devel] [PATCH] 304 Allow localhost in zone ACIs In-Reply-To: <504E3B83.4020308@redhat.com> References: <50487215.70600@redhat.com> <50488A8E.4030409@redhat.com> <5048904F.6060802@redhat.com> <504E3B83.4020308@redhat.com> Message-ID: <504F5594.3030701@redhat.com> On 09/10/2012 09:12 PM, Endi Sukma Dewata wrote: > On 9/6/2012 7:00 AM, Martin Kosek wrote: >> On 09/06/2012 01:35 PM, Petr Vobornik wrote: >>> On 09/06/2012 11:51 AM, Martin Kosek wrote: >>>> Loopback address, "localhost" and "localnets" ACIs are no longer >>>> an issue for bind-dyndb-ldap. Allow them in our validators. >>>> >>> >>> Martin's patch works and looks good - ACK. >>> >>> >>> Attaching patch for Web UI part. >> >> Web UI validator works fine too, ACK. >> >> Pushed both patches to master, ipa-3-0. >> >> Martin > > Looks like there's a problem in DNS zone details page. The > idnssoaserial's maxvalue is an empty string so the validation fails even > if you are just opening an existing DNS zone. So either the server > should not return maxvalue or the UI should check for possible empty > string. > I fixed the problem on both sides in patches 212, 213. 212 is fixing Web UI to be able to work with empty strings. 213 is fixing python part: serialization of long -- Petr Vobornik From rcritten at redhat.com Tue Sep 11 15:19:59 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 11 Sep 2012 11:19:59 -0400 Subject: [Freeipa-devel] [PATCH] 1054 new default SELinux user map user In-Reply-To: <20120910215402.GE11130@hendrix.redhat.com> References: <504E5DE7.804@redhat.com> <20120910215402.GE11130@hendrix.redhat.com> Message-ID: <504F569F.6000002@redhat.com> Jakub Hrozek wrote: > On Mon, Sep 10, 2012 at 05:38:47PM -0400, Rob Crittenden wrote: >> We've decided to change the default SELinux user map user to the OS >> default which is unconfined_u. It would be too drastic to go from >> one extreme to another. >> >> rob > > How does one set an "empty default" that the SSSD would treat as "don't > create any login file whatsoever" ? Patch updated to support that. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-rcrit-1054-2-selinux.patch Type: text/x-diff Size: 3503 bytes Desc: not available URL: From pviktori at redhat.com Tue Sep 11 16:33:09 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Tue, 11 Sep 2012 18:33:09 +0200 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <504F4CE0.9080706@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502B86F5.4070008@redhat.com> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> <1347329065.24731.101.camel@aleeredhat.laptop> <504F318F.3030603@redhat.com> <504F35AD.3030905@redhat.com> <1347373691.24731.125.camel@aleeredhat.laptop> <504F4CE0.9080706@redhat.com> Message-ID: <504F67C5.4000600@redhat.com> On 09/11/2012 04:38 PM, Rob Crittenden wrote: > Ade Lee wrote: >> On Tue, 2012-09-11 at 08:59 -0400, Rob Crittenden wrote: >>> Petr Viktorin wrote: >>>> On 09/11/2012 04:04 AM, Ade Lee wrote: >>>>> On Mon, 2012-09-10 at 16:58 -0400, Rob Crittenden wrote: >>>>>> Petr Viktorin wrote: >>>>>>> Attaching rebased and squashed patches. I've done some testing with >>>>>>> them >>>>>>> but please test some more. >>>>>>> >>>>>> >>>>>> Most of these aren't IPA issues, but dogtag issues. I'll try to split >>>>>> them out. >>>>>> >>>>>> IPA: >>>>>> >>>>>> For the configuration files in install/conf to be updated at rpm >>>>>> update >>>>>> time the VERSION needs to be incremented. >>>> >>>> These files should stay the same since on upgrade we're still using a >>>> Dogtag 9 style instance. The Dogtag 10 ports are only used in new >>>> installs. >>>> >>>>>> The ipa package lacks any updated dogtag dependencies, so I abused >>>>>> it. >>>> >>>> What should the updated dependencies be? Since it should work with both >>>> dogtag 9 and 10, I don't see how they should change. >>> >>> I don't know either, but we need to prevent people from installing >>> incompatible package combinations. >>> >> Would'nt the Conflicts: ipa < 3.0 in pki-ca mentioned below satisfy this >> requirement? The main concern is that you must have ipa 3.0 if you have >> dogtag 10. >> >> Given that dogtag is consumed by IPA though, it makes more sense to put >> the relevant conflicts in IPA rather than in dogtag. So in this case, >> that would mean putting Conflicts: pki-ca >= 10.0 in IPA 2.x. >> Recall that dogtag 10 will only be officially available in f18+. > > That isn't enough. If a F-17 user with IPA 2.2 installed upgrades to > F-18 they would be able to install dogtag 10 and blow up their IPA server. > >> >>>>>> I installed IPA with dogtag 9 and created a replica. >>>>>> >>>>>> I updated the IPA bits, that worked fine. >>>>>> >>>>>> I updated to dogtag 10 and now the CA doesn't work on the master, >>>>>> including starting the dogtag instance. Note that the rpm update >>>>>> process >>>>>> worked, no notice that the CA service didn't restart. >>>>>> >>>>> Did you try to restart the CA with selinux in permissive mode? >>>>> This is >>>>> still required right now until I get the selinux policy all >>>>> straightened >>>>> out. >>>>> >>>>> There is also a separate dogtag ticket (which is currently being >>>>> worked >>>>> on) to restart existing dogtag instances when dogtag is upgraded from >>>>> 9->10. >>>> >>>> In permissive mode, this upgrade works for me. >>> >>> I was in enforcing mode but saw no AVCs. What is the ETA on fixing this? >>> >> >> Within the next week or two, I need to finish the IPA merge database >> patch first, and then co-ordinate changes with the selinux guys. >> >>>> >>>> >>>> Sometimes I do get this error intermittently: >>>> >>>> ipa: ERROR: Certificate operation cannot be completed: Unable to >>>> communicate with CMS (Service Temporarily Unavailable) >>>> >>>> Usually, waiting a couple of minutes clears this up. Perhaps we are not >>>> doing startup detection correctly. Ade mentioned that waiting for ports >>>> may not be ideal. How do we know if Dogtag is initialized? >>> >>> Years ago we had discussed with Andrew and Matt creating a URI that can >>> be queried to determine dogtag status. I don't think that ever went >>> anywhere. >>> >> Petr, this happens only on reboot, right? And not on regular "service >> ipa restart"? I've now seen it happen right after a 9 ? 10 upgrade. >> Yeah, I remember this conversation - and even created a bug for it at >> some point. This went away because the mechanism you were using seemed >> to be working. The timing may be very different now with tomcat 7 and >> systemd. I'll open a dogtag trac ticket for this. > > Ok. > >> >>>> >>>>>> Uninstalling failed because it tried to run pkidestroy and not >>>>>> pkiremove. >>>> >>>> I was under the impression that pkidestroy was the correct command to >>>> remove an upgraded instance. I'll check with Ade. >>>> >>>>> I'll test this too. >>>>> >>>>>> The contents of the file passed to pkispawn should be logged so we >>>>>> can >>>>>> see exactly what was passed in. >>>>>> >>>>> Its a pretty big file. You might want to only log the modifications. >>>>> Or save the file somewhere. >>>> >>>> Our logs are pretty verbose, so that shouldn't be a problem. I'll >>>> put it >>>> in the next version of the patch. >>> >>> The question to ask is: would you need the contents of this file if all >>> you got were logs and needed to evaluate why installation failed? In >>> most cases this is yes. >>> >> Up to you guys. There is a patch I am working on in which I will be >> logging the object that is being passed to the server from pkispawn. >> That - and the diffs to the standard config file as I mentioned above - >> will likely be sufficient to debug almost all cases. >> >> Make sure not to log any passwords. >> Thanks for the catch. Attaching updated patch that sanitizes the passwords. >>>>>> DOGTAG: >>>>>> >>>>>> When upgrading using the dogtag-devel repo I had to specify >>>>>> pki-tools.x86_64 otherwise it tried to install both 32 and 64-bit >>>>>> versions (and failed). >>>>>> >>>>>> I ended up running: yum update pki-ca tomcatjss pki-tools.x86_64 >>>>>> --enablerepo=dogtag-devel --enablerepo=updates-testing >>>>>> >>>>> We'll look into this. I think I know why this happens. >>>>> >>>>>> What happens if someone manually upgrades pki-ca without first >>>>>> updating >>>>>> ipa? I think that pki-ca is going to need a Conflicts ipa < 3.0 in >>>>>> it. >>>>> >>>>> We can add that. >>>>> >>>>>> certificate renewal failed. I spent far too long trying to figure out >>>>>> why tomcat wasn't listening on port 9180 but failed. I think 9180 is >>>>>> actually the old server, right? So another missing dependency on a >>>>>> fixed >>>>>> certmonger? >>>>>> >>>>>> The best I could find was the certmonger error: >>>>>> >>>>>> ca-error: Error 7 connecting to >>>>>> http://edsel.example.com:9180/ca/ee/ca/profileSubmit: Couldn't >>>>>> connect >>>>>> to server. >>>>>> >>>> >>>> I'll set the certmonger min version to 0.60 as per Nalin's mail. >>> >>> Ok. >>> >>>>> Is this cert renewal on a dogtag 10 instance? Or the upgraded >>>>> dogtag 9? >>>>> If its dogtag 10, perhaps you do not have the certmonger version that >>>>> has the relevant fix? If its dogtag 9, then we need to figure out >>>>> whats >>>>> going on. That reminds me - I need to file a bug to allow >>>>> certmonger to >>>>> talk to the newly defined dogtag ports. Do you have selinux >>>>> permissive? >>>>> >>>>>> There is no man page for pkispawn/pkidestroy :-( According to the FHS >>>>>> these should not be in /bin but in /usr/sbin (not end-user commands). >>>>>> >>>>> There is a trac ticket open for man pages for pkispawn and pkidestroy. >>>>> We plan to complete this ticket by the time f18 is released. >>>>> >>>>> We'll look into the location of pkispawn/pkicreate. >>>>> >>>>>> The output of pkicreate/pkisilent was really terrible and not >>>>>> usable at >>>>>> all so we didn't display it when failures occurred. It looks like >>>>>> that >>>>>> has been addressed, at least for the case where a CA is already >>>>>> configured and you try to install IPA. Perhaps we should capture >>>>>> stderr >>>>>> and display that instead of the command-line of pkispawn? Again, a >>>>>> man >>>>>> page would help with the integration. >>>>>> >>>>>> 2012-09-10T20:51:45Z DEBUG [2/18]: configuring certificate server >>>>>> instance >>>>>> 2012-09-10T20:51:45Z DEBUG args=/bin/pkispawn -s CA -f /tmp/tmp_Urraq >>>>>> 2012-09-10T20:51:45Z DEBUG stdout= >>>>>> 2012-09-10T20:51:45Z DEBUG stderr=pkispawn : ERROR ....... PKI >>>>>> subsystem 'CA' for instance 'pki-tomcat' already exists! >>>>>> >>>>>> 2012-09-10T20:51:45Z CRITICAL failed to configure ca instance Command >>>>>> '/bin/pkispawn -s CA -f /tmp/tmp_Urraq' returned non-zero exit >>>>>> status 1 >>>>>> >>>>> That may be a good idea. Of course. thats an IPA thing, right? >>> >>> Right, not a show-stopper for this but a new ticket should be opened to >>> track it. https://fedorahosted.org/freeipa/ticket/3072 -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0074-06-Use-Dogtag-10-only-when-it-is-available.patch Type: text/x-patch Size: 59768 bytes Desc: not available URL: From mkosek at redhat.com Tue Sep 11 16:51:34 2012 From: mkosek at redhat.com (Martin Kosek) Date: Tue, 11 Sep 2012 18:51:34 +0200 Subject: [Freeipa-devel] Ticket #2866 - referential integrity in IPA In-Reply-To: <504612DC.9060402@redhat.com> References: <1345211089.3796.34.camel@priserak> <502E4E78.1090407@redhat.com> <503B9F4E.4010005@redhat.com> <503BA065.1060207@redhat.com> <5044C1F0.7080609@redhat.com> <504612DC.9060402@redhat.com> Message-ID: <504F6C16.30400@redhat.com> On 09/04/2012 04:40 PM, Rich Megginson wrote: > On 09/03/2012 08:42 AM, Martin Kosek wrote: >> On 08/27/2012 06:29 PM, Rich Megginson wrote: ... >> This is the plan I plan to take: >> 1) Add pres,eq indexes for all un-indexed attributes that we want to check: >> sourcehost >> memberservice >> managedby >> memberallowcmd >> memberdenycmd >> ipasudorunas >> ipasudorunasgroup > ok ... Implementation of the Referential Integrity in IPA works OK so far, I just hit a strange issue when indexing memberallowcmd and memberdenycmd attributes in IPA: dirsrv errors log: ... [11/Sep/2012:11:39:53 -0400] - The attribute [memberdenycmd] does not have a valid ORDERING matching rule - error 2:s [11/Sep/2012:11:39:58 -0400] - userRoot: Indexing attribute: memberdenycmd [11/Sep/2012:11:39:58 -0400] - userRoot: Finished indexing. ... This cause RI to fail to handle this attribute in IPA (which is expected based on the error message). I checked the attributes types, they look like that: attributetypes: ( 2.16.840.1.113730.3.8.7.1 NAME 'memberAllowCmd' DESC 'Refere nce to a command or group of commands that are allowed by the rule.' SUP dist inguishedName EQUALITY distinguishedNameMatch ORDERING distinguishedNameMatch SUBSTR distinguishedNameMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.12 X-ORIGIN 'IPA v2' ) attributetypes: ( 2.16.840.1.113730.3.8.7.2 NAME 'memberDenyCmd' DESC 'Referen ce to a command or group of commands that are denied by the rule.' SUP distin guishedName EQUALITY distinguishedNameMatch ORDERING distinguishedNameMatch S UBSTR distinguishedNameMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.12 X-ORIGIN 'I PA v2' ) "distinguishedNameMatch" ORDERING rule looks wrong, it is only a matching rule. Anyone knows why it was defined that way? Or what should be a correct setting for this attribute instead? caseIgnoreOrderingMatch seems as a logical choice... I will have to fix the attributeTypes definition in IPA before we can enable index & RI for them. Thanks, Martin From rcritten at redhat.com Tue Sep 11 18:45:20 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 11 Sep 2012 14:45:20 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <504F67C5.4000600@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502B86F5.4070008@redhat.com> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> <1347329065.24731.101.camel@aleeredhat.laptop> <504F318F.3030603@redhat.com> <504F35AD.3030905@redhat.com> <1347373691.24731.125.camel@aleeredhat.laptop> <504F4CE0.9080706@redhat.com> <504F67C5.4000600@redhat.com> Message-ID: <504F86C0.6060907@redhat.com> Petr Viktorin wrote: > On 09/11/2012 04:38 PM, Rob Crittenden wrote: >> Ade Lee wrote: >>> On Tue, 2012-09-11 at 08:59 -0400, Rob Crittenden wrote: >>>> Petr Viktorin wrote: >>>>> On 09/11/2012 04:04 AM, Ade Lee wrote: >>>>>> On Mon, 2012-09-10 at 16:58 -0400, Rob Crittenden wrote: >>>>>>> Petr Viktorin wrote: >>>>>>>> Attaching rebased and squashed patches. I've done some testing with >>>>>>>> them >>>>>>>> but please test some more. >>>>>>>> >>>>>>> >>>>>>> Most of these aren't IPA issues, but dogtag issues. I'll try to >>>>>>> split >>>>>>> them out. >>>>>>> >>>>>>> IPA: >>>>>>> >>>>>>> For the configuration files in install/conf to be updated at rpm >>>>>>> update >>>>>>> time the VERSION needs to be incremented. >>>>> >>>>> These files should stay the same since on upgrade we're still using a >>>>> Dogtag 9 style instance. The Dogtag 10 ports are only used in new >>>>> installs. >>>>> >>>>>>> The ipa package lacks any updated dogtag dependencies, so I abused >>>>>>> it. >>>>> >>>>> What should the updated dependencies be? Since it should work with >>>>> both >>>>> dogtag 9 and 10, I don't see how they should change. >>>> >>>> I don't know either, but we need to prevent people from installing >>>> incompatible package combinations. >>>> >>> Would'nt the Conflicts: ipa < 3.0 in pki-ca mentioned below satisfy this >>> requirement? The main concern is that you must have ipa 3.0 if you have >>> dogtag 10. >>> >>> Given that dogtag is consumed by IPA though, it makes more sense to put >>> the relevant conflicts in IPA rather than in dogtag. So in this case, >>> that would mean putting Conflicts: pki-ca >= 10.0 in IPA 2.x. >>> Recall that dogtag 10 will only be officially available in f18+. >> >> That isn't enough. If a F-17 user with IPA 2.2 installed upgrades to >> F-18 they would be able to install dogtag 10 and blow up their IPA >> server. >> >>> >>>>>>> I installed IPA with dogtag 9 and created a replica. >>>>>>> >>>>>>> I updated the IPA bits, that worked fine. >>>>>>> >>>>>>> I updated to dogtag 10 and now the CA doesn't work on the master, >>>>>>> including starting the dogtag instance. Note that the rpm update >>>>>>> process >>>>>>> worked, no notice that the CA service didn't restart. >>>>>>> >>>>>> Did you try to restart the CA with selinux in permissive mode? >>>>>> This is >>>>>> still required right now until I get the selinux policy all >>>>>> straightened >>>>>> out. >>>>>> >>>>>> There is also a separate dogtag ticket (which is currently being >>>>>> worked >>>>>> on) to restart existing dogtag instances when dogtag is upgraded from >>>>>> 9->10. >>>>> >>>>> In permissive mode, this upgrade works for me. >>>> >>>> I was in enforcing mode but saw no AVCs. What is the ETA on fixing >>>> this? >>>> >>> >>> Within the next week or two, I need to finish the IPA merge database >>> patch first, and then co-ordinate changes with the selinux guys. >>> >>>>> >>>>> >>>>> Sometimes I do get this error intermittently: >>>>> >>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to >>>>> communicate with CMS (Service Temporarily Unavailable) >>>>> >>>>> Usually, waiting a couple of minutes clears this up. Perhaps we are >>>>> not >>>>> doing startup detection correctly. Ade mentioned that waiting for >>>>> ports >>>>> may not be ideal. How do we know if Dogtag is initialized? >>>> >>>> Years ago we had discussed with Andrew and Matt creating a URI that can >>>> be queried to determine dogtag status. I don't think that ever went >>>> anywhere. >>>> >>> Petr, this happens only on reboot, right? And not on regular "service >>> ipa restart"? > > I've now seen it happen right after a 9 ? 10 upgrade. > >>> Yeah, I remember this conversation - and even created a bug for it at >>> some point. This went away because the mechanism you were using seemed >>> to be working. The timing may be very different now with tomcat 7 and >>> systemd. I'll open a dogtag trac ticket for this. >> >> Ok. >> >>> >>>>> >>>>>>> Uninstalling failed because it tried to run pkidestroy and not >>>>>>> pkiremove. >>>>> >>>>> I was under the impression that pkidestroy was the correct command to >>>>> remove an upgraded instance. I'll check with Ade. >>>>> >>>>>> I'll test this too. >>>>>> >>>>>>> The contents of the file passed to pkispawn should be logged so we >>>>>>> can >>>>>>> see exactly what was passed in. >>>>>>> >>>>>> Its a pretty big file. You might want to only log the modifications. >>>>>> Or save the file somewhere. >>>>> >>>>> Our logs are pretty verbose, so that shouldn't be a problem. I'll >>>>> put it >>>>> in the next version of the patch. >>>> >>>> The question to ask is: would you need the contents of this file if all >>>> you got were logs and needed to evaluate why installation failed? In >>>> most cases this is yes. >>>> >>> Up to you guys. There is a patch I am working on in which I will be >>> logging the object that is being passed to the server from pkispawn. >>> That - and the diffs to the standard config file as I mentioned above - >>> will likely be sufficient to debug almost all cases. >>> >>> Make sure not to log any passwords. >>> > > Thanks for the catch. Attaching updated patch that sanitizes the passwords. > >>>>>>> DOGTAG: >>>>>>> >>>>>>> When upgrading using the dogtag-devel repo I had to specify >>>>>>> pki-tools.x86_64 otherwise it tried to install both 32 and 64-bit >>>>>>> versions (and failed). >>>>>>> >>>>>>> I ended up running: yum update pki-ca tomcatjss pki-tools.x86_64 >>>>>>> --enablerepo=dogtag-devel --enablerepo=updates-testing >>>>>>> >>>>>> We'll look into this. I think I know why this happens. >>>>>> >>>>>>> What happens if someone manually upgrades pki-ca without first >>>>>>> updating >>>>>>> ipa? I think that pki-ca is going to need a Conflicts ipa < 3.0 in >>>>>>> it. >>>>>> >>>>>> We can add that. >>>>>> >>>>>>> certificate renewal failed. I spent far too long trying to figure >>>>>>> out >>>>>>> why tomcat wasn't listening on port 9180 but failed. I think 9180 is >>>>>>> actually the old server, right? So another missing dependency on a >>>>>>> fixed >>>>>>> certmonger? >>>>>>> >>>>>>> The best I could find was the certmonger error: >>>>>>> >>>>>>> ca-error: Error 7 connecting to >>>>>>> http://edsel.example.com:9180/ca/ee/ca/profileSubmit: Couldn't >>>>>>> connect >>>>>>> to server. >>>>>>> >>>>> >>>>> I'll set the certmonger min version to 0.60 as per Nalin's mail. >>>> >>>> Ok. >>>> >>>>>> Is this cert renewal on a dogtag 10 instance? Or the upgraded >>>>>> dogtag 9? >>>>>> If its dogtag 10, perhaps you do not have the certmonger version that >>>>>> has the relevant fix? If its dogtag 9, then we need to figure out >>>>>> whats >>>>>> going on. That reminds me - I need to file a bug to allow >>>>>> certmonger to >>>>>> talk to the newly defined dogtag ports. Do you have selinux >>>>>> permissive? >>>>>> >>>>>>> There is no man page for pkispawn/pkidestroy :-( According to the >>>>>>> FHS >>>>>>> these should not be in /bin but in /usr/sbin (not end-user >>>>>>> commands). >>>>>>> >>>>>> There is a trac ticket open for man pages for pkispawn and >>>>>> pkidestroy. >>>>>> We plan to complete this ticket by the time f18 is released. >>>>>> >>>>>> We'll look into the location of pkispawn/pkicreate. >>>>>> >>>>>>> The output of pkicreate/pkisilent was really terrible and not >>>>>>> usable at >>>>>>> all so we didn't display it when failures occurred. It looks like >>>>>>> that >>>>>>> has been addressed, at least for the case where a CA is already >>>>>>> configured and you try to install IPA. Perhaps we should capture >>>>>>> stderr >>>>>>> and display that instead of the command-line of pkispawn? Again, a >>>>>>> man >>>>>>> page would help with the integration. >>>>>>> >>>>>>> 2012-09-10T20:51:45Z DEBUG [2/18]: configuring certificate server >>>>>>> instance >>>>>>> 2012-09-10T20:51:45Z DEBUG args=/bin/pkispawn -s CA -f >>>>>>> /tmp/tmp_Urraq >>>>>>> 2012-09-10T20:51:45Z DEBUG stdout= >>>>>>> 2012-09-10T20:51:45Z DEBUG stderr=pkispawn : ERROR ....... PKI >>>>>>> subsystem 'CA' for instance 'pki-tomcat' already exists! >>>>>>> >>>>>>> 2012-09-10T20:51:45Z CRITICAL failed to configure ca instance >>>>>>> Command >>>>>>> '/bin/pkispawn -s CA -f /tmp/tmp_Urraq' returned non-zero exit >>>>>>> status 1 >>>>>>> >>>>>> That may be a good idea. Of course. thats an IPA thing, right? >>>> >>>> Right, not a show-stopper for this but a new ticket should be opened to >>>> track it. > > https://fedorahosted.org/freeipa/ticket/3072 Thanks. I tested a 2.2 -> 3.0 upgrade with dogtag 9 and it seems to have failed to restart something: [ post rpm -Uvh dist/rpms/freeipa*.rpm ] [root at edsel freeipa]# ipa cert-show 1 ipa: ERROR: Certificate operation cannot be completed: Unable to communicate with CMS (Service Temporarily Unavailable) [root at edsel freeipa]# ipactl restart Restarting Directory Service Restarting KDC Service Restarting KPASSWD Service Restarting MEMCACHE Service Restarting HTTP Service Restarting CA Service [root at edsel freeipa]# ipa cert-show 1 Certificate: MIIDjTCCAnWgAwIBAgIBATANBgk ... The apache log had this: [Tue Sep 11 14:35:42 2012] [error] (111)Connection refused: proxy: AJP: attempt to connect to 127.0.0.1:9447 (localhost) failed [Tue Sep 11 14:35:42 2012] [error] ap_proxy_connect_backend disabling worker for (localhost) [Tue Sep 11 14:35:42 2012] [error] proxy: AJP: failed to make connection to backend: localhost So I'm gathering that dogtag didn't restart properly, but I'm just guessing. It could have been the PKI-IPA ds instance too, I'm not sure where to check. I also noticed this in /var/log/ipaupgrade.log: 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl start certmonger.service 2012-09-11T18:28:22Z DEBUG stdout= 2012-09-11T18:28:22Z DEBUG stderr= 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl is-active certmonger.service 2012-09-11T18:28:22Z DEBUG stdout=active ... ... [ snip certutil output ] ... 2012-09-11T18:28:52Z DEBUG args=/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" -P XXXXXXXX 2012-09-11T18:28:52Z DEBUG stdout=Please verify that the certmonger service is still running. 2012-09-11T18:28:52Z DEBUG stderr= 2012-09-11T18:28:52Z INFO File "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", line 614, in run_script return_value = main_function() File "/usr/sbin/ipa-upgradeconfig", line 540, in main enable_certificate_renewal(api.env.realm) File "/usr/sbin/ipa-upgradeconfig", line 455, in enable_certificate_renewal ca.configure_renewal() File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1298, in configure_renewal self.dogtag_constants.ALIAS_DIR, 'renew_ca_cert "%s"' % nickname) File "/usr/lib/python2.7/site-packages/ipapython/certmonger.py", line 394, in dogtag_start_tracking (stdout, stderr, returncode) = ipautil.run(args, nolog=[pin]) File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 309, in run raise CalledProcessError(p.returncode, args) 2012-09-11T18:28:52Z INFO The ipa-upgradeconfig command failed, exception: CalledProcessError: Command '/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit status 1 I'm not sure if this is related to this patch or not. If it isn't can you file a new ticket on it, or add it the 2.2 -> 3.0 upgrade ticket? rob From rcritten at redhat.com Tue Sep 11 19:20:24 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 11 Sep 2012 15:20:24 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <504F86C0.6060907@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502B86F5.4070008@redhat.com> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> <1347329065.24731.101.camel@aleeredhat.laptop> <504F318F.3030603@redhat.com> <504F35AD.3030905@redhat.com> <1347373691.24731.125.camel@aleeredhat.laptop> <504F4CE0.9080706@redhat.com> <504F67C5.4000600@redhat.com> <504F86C0.6060907@redhat.com> Message-ID: <504F8EF8.2010601@redhat.com> Rob Crittenden wrote: > Petr Viktorin wrote: >> On 09/11/2012 04:38 PM, Rob Crittenden wrote: >>> Ade Lee wrote: >>>> On Tue, 2012-09-11 at 08:59 -0400, Rob Crittenden wrote: >>>>> Petr Viktorin wrote: >>>>>> On 09/11/2012 04:04 AM, Ade Lee wrote: >>>>>>> On Mon, 2012-09-10 at 16:58 -0400, Rob Crittenden wrote: >>>>>>>> Petr Viktorin wrote: >>>>>>>>> Attaching rebased and squashed patches. I've done some testing >>>>>>>>> with >>>>>>>>> them >>>>>>>>> but please test some more. >>>>>>>>> >>>>>>>> >>>>>>>> Most of these aren't IPA issues, but dogtag issues. I'll try to >>>>>>>> split >>>>>>>> them out. >>>>>>>> >>>>>>>> IPA: >>>>>>>> >>>>>>>> For the configuration files in install/conf to be updated at rpm >>>>>>>> update >>>>>>>> time the VERSION needs to be incremented. >>>>>> >>>>>> These files should stay the same since on upgrade we're still using a >>>>>> Dogtag 9 style instance. The Dogtag 10 ports are only used in new >>>>>> installs. >>>>>> >>>>>>>> The ipa package lacks any updated dogtag dependencies, so I abused >>>>>>>> it. >>>>>> >>>>>> What should the updated dependencies be? Since it should work with >>>>>> both >>>>>> dogtag 9 and 10, I don't see how they should change. >>>>> >>>>> I don't know either, but we need to prevent people from installing >>>>> incompatible package combinations. >>>>> >>>> Would'nt the Conflicts: ipa < 3.0 in pki-ca mentioned below satisfy >>>> this >>>> requirement? The main concern is that you must have ipa 3.0 if you >>>> have >>>> dogtag 10. >>>> >>>> Given that dogtag is consumed by IPA though, it makes more sense to put >>>> the relevant conflicts in IPA rather than in dogtag. So in this case, >>>> that would mean putting Conflicts: pki-ca >= 10.0 in IPA 2.x. >>>> Recall that dogtag 10 will only be officially available in f18+. >>> >>> That isn't enough. If a F-17 user with IPA 2.2 installed upgrades to >>> F-18 they would be able to install dogtag 10 and blow up their IPA >>> server. >>> >>>> >>>>>>>> I installed IPA with dogtag 9 and created a replica. >>>>>>>> >>>>>>>> I updated the IPA bits, that worked fine. >>>>>>>> >>>>>>>> I updated to dogtag 10 and now the CA doesn't work on the master, >>>>>>>> including starting the dogtag instance. Note that the rpm update >>>>>>>> process >>>>>>>> worked, no notice that the CA service didn't restart. >>>>>>>> >>>>>>> Did you try to restart the CA with selinux in permissive mode? >>>>>>> This is >>>>>>> still required right now until I get the selinux policy all >>>>>>> straightened >>>>>>> out. >>>>>>> >>>>>>> There is also a separate dogtag ticket (which is currently being >>>>>>> worked >>>>>>> on) to restart existing dogtag instances when dogtag is upgraded >>>>>>> from >>>>>>> 9->10. >>>>>> >>>>>> In permissive mode, this upgrade works for me. >>>>> >>>>> I was in enforcing mode but saw no AVCs. What is the ETA on fixing >>>>> this? >>>>> >>>> >>>> Within the next week or two, I need to finish the IPA merge database >>>> patch first, and then co-ordinate changes with the selinux guys. >>>> >>>>>> >>>>>> >>>>>> Sometimes I do get this error intermittently: >>>>>> >>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to >>>>>> communicate with CMS (Service Temporarily Unavailable) >>>>>> >>>>>> Usually, waiting a couple of minutes clears this up. Perhaps we are >>>>>> not >>>>>> doing startup detection correctly. Ade mentioned that waiting for >>>>>> ports >>>>>> may not be ideal. How do we know if Dogtag is initialized? >>>>> >>>>> Years ago we had discussed with Andrew and Matt creating a URI that >>>>> can >>>>> be queried to determine dogtag status. I don't think that ever went >>>>> anywhere. >>>>> >>>> Petr, this happens only on reboot, right? And not on regular "service >>>> ipa restart"? >> >> I've now seen it happen right after a 9 ? 10 upgrade. >> >>>> Yeah, I remember this conversation - and even created a bug for it at >>>> some point. This went away because the mechanism you were using seemed >>>> to be working. The timing may be very different now with tomcat 7 and >>>> systemd. I'll open a dogtag trac ticket for this. >>> >>> Ok. >>> >>>> >>>>>> >>>>>>>> Uninstalling failed because it tried to run pkidestroy and not >>>>>>>> pkiremove. >>>>>> >>>>>> I was under the impression that pkidestroy was the correct command to >>>>>> remove an upgraded instance. I'll check with Ade. >>>>>> >>>>>>> I'll test this too. >>>>>>> >>>>>>>> The contents of the file passed to pkispawn should be logged so we >>>>>>>> can >>>>>>>> see exactly what was passed in. >>>>>>>> >>>>>>> Its a pretty big file. You might want to only log the >>>>>>> modifications. >>>>>>> Or save the file somewhere. >>>>>> >>>>>> Our logs are pretty verbose, so that shouldn't be a problem. I'll >>>>>> put it >>>>>> in the next version of the patch. >>>>> >>>>> The question to ask is: would you need the contents of this file if >>>>> all >>>>> you got were logs and needed to evaluate why installation failed? In >>>>> most cases this is yes. >>>>> >>>> Up to you guys. There is a patch I am working on in which I will be >>>> logging the object that is being passed to the server from pkispawn. >>>> That - and the diffs to the standard config file as I mentioned above - >>>> will likely be sufficient to debug almost all cases. >>>> >>>> Make sure not to log any passwords. >>>> >> >> Thanks for the catch. Attaching updated patch that sanitizes the >> passwords. >> >>>>>>>> DOGTAG: >>>>>>>> >>>>>>>> When upgrading using the dogtag-devel repo I had to specify >>>>>>>> pki-tools.x86_64 otherwise it tried to install both 32 and 64-bit >>>>>>>> versions (and failed). >>>>>>>> >>>>>>>> I ended up running: yum update pki-ca tomcatjss pki-tools.x86_64 >>>>>>>> --enablerepo=dogtag-devel --enablerepo=updates-testing >>>>>>>> >>>>>>> We'll look into this. I think I know why this happens. >>>>>>> >>>>>>>> What happens if someone manually upgrades pki-ca without first >>>>>>>> updating >>>>>>>> ipa? I think that pki-ca is going to need a Conflicts ipa < 3.0 in >>>>>>>> it. >>>>>>> >>>>>>> We can add that. >>>>>>> >>>>>>>> certificate renewal failed. I spent far too long trying to figure >>>>>>>> out >>>>>>>> why tomcat wasn't listening on port 9180 but failed. I think >>>>>>>> 9180 is >>>>>>>> actually the old server, right? So another missing dependency on a >>>>>>>> fixed >>>>>>>> certmonger? >>>>>>>> >>>>>>>> The best I could find was the certmonger error: >>>>>>>> >>>>>>>> ca-error: Error 7 connecting to >>>>>>>> http://edsel.example.com:9180/ca/ee/ca/profileSubmit: Couldn't >>>>>>>> connect >>>>>>>> to server. >>>>>>>> >>>>>> >>>>>> I'll set the certmonger min version to 0.60 as per Nalin's mail. >>>>> >>>>> Ok. >>>>> >>>>>>> Is this cert renewal on a dogtag 10 instance? Or the upgraded >>>>>>> dogtag 9? >>>>>>> If its dogtag 10, perhaps you do not have the certmonger version >>>>>>> that >>>>>>> has the relevant fix? If its dogtag 9, then we need to figure out >>>>>>> whats >>>>>>> going on. That reminds me - I need to file a bug to allow >>>>>>> certmonger to >>>>>>> talk to the newly defined dogtag ports. Do you have selinux >>>>>>> permissive? >>>>>>> >>>>>>>> There is no man page for pkispawn/pkidestroy :-( According to the >>>>>>>> FHS >>>>>>>> these should not be in /bin but in /usr/sbin (not end-user >>>>>>>> commands). >>>>>>>> >>>>>>> There is a trac ticket open for man pages for pkispawn and >>>>>>> pkidestroy. >>>>>>> We plan to complete this ticket by the time f18 is released. >>>>>>> >>>>>>> We'll look into the location of pkispawn/pkicreate. >>>>>>> >>>>>>>> The output of pkicreate/pkisilent was really terrible and not >>>>>>>> usable at >>>>>>>> all so we didn't display it when failures occurred. It looks like >>>>>>>> that >>>>>>>> has been addressed, at least for the case where a CA is already >>>>>>>> configured and you try to install IPA. Perhaps we should capture >>>>>>>> stderr >>>>>>>> and display that instead of the command-line of pkispawn? Again, a >>>>>>>> man >>>>>>>> page would help with the integration. >>>>>>>> >>>>>>>> 2012-09-10T20:51:45Z DEBUG [2/18]: configuring certificate server >>>>>>>> instance >>>>>>>> 2012-09-10T20:51:45Z DEBUG args=/bin/pkispawn -s CA -f >>>>>>>> /tmp/tmp_Urraq >>>>>>>> 2012-09-10T20:51:45Z DEBUG stdout= >>>>>>>> 2012-09-10T20:51:45Z DEBUG stderr=pkispawn : ERROR ....... >>>>>>>> PKI >>>>>>>> subsystem 'CA' for instance 'pki-tomcat' already exists! >>>>>>>> >>>>>>>> 2012-09-10T20:51:45Z CRITICAL failed to configure ca instance >>>>>>>> Command >>>>>>>> '/bin/pkispawn -s CA -f /tmp/tmp_Urraq' returned non-zero exit >>>>>>>> status 1 >>>>>>>> >>>>>>> That may be a good idea. Of course. thats an IPA thing, right? >>>>> >>>>> Right, not a show-stopper for this but a new ticket should be >>>>> opened to >>>>> track it. >> >> https://fedorahosted.org/freeipa/ticket/3072 > > Thanks. > > I tested a 2.2 -> 3.0 upgrade with dogtag 9 and it seems to have failed > to restart something: > > [ post rpm -Uvh dist/rpms/freeipa*.rpm ] > > [root at edsel freeipa]# ipa cert-show 1 > ipa: ERROR: Certificate operation cannot be completed: Unable to > communicate with CMS (Service Temporarily Unavailable) > [root at edsel freeipa]# ipactl restart > Restarting Directory Service > Restarting KDC Service > Restarting KPASSWD Service > Restarting MEMCACHE Service > Restarting HTTP Service > Restarting CA Service > [root at edsel freeipa]# ipa cert-show 1 > Certificate: MIIDjTCCAnWgAwIBAgIBATANBgk > ... > > The apache log had this: > > [Tue Sep 11 14:35:42 2012] [error] (111)Connection refused: proxy: AJP: > attempt to connect to 127.0.0.1:9447 (localhost) failed > [Tue Sep 11 14:35:42 2012] [error] ap_proxy_connect_backend disabling > worker for (localhost) > [Tue Sep 11 14:35:42 2012] [error] proxy: AJP: failed to make connection > to backend: localhost > > So I'm gathering that dogtag didn't restart properly, but I'm just > guessing. It could have been the PKI-IPA ds instance too, I'm not sure > where to check. > > I also noticed this in /var/log/ipaupgrade.log: > > 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl start certmonger.service > 2012-09-11T18:28:22Z DEBUG stdout= > 2012-09-11T18:28:22Z DEBUG stderr= > 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl is-active certmonger.service > 2012-09-11T18:28:22Z DEBUG stdout=active > ... > ... [ snip certutil output ] > ... > 2012-09-11T18:28:52Z DEBUG args=/usr/bin/getcert start-tracking -d > /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c > dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert > "auditSigningCert cert-pki-ca" -P XXXXXXXX > 2012-09-11T18:28:52Z DEBUG stdout=Please verify that the certmonger > service is still running. > > 2012-09-11T18:28:52Z DEBUG stderr= > 2012-09-11T18:28:52Z INFO File > "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", > line 614, in run_script > return_value = main_function() > > File "/usr/sbin/ipa-upgradeconfig", line 540, in main > enable_certificate_renewal(api.env.realm) > > File "/usr/sbin/ipa-upgradeconfig", line 455, in > enable_certificate_renewal > ca.configure_renewal() > > File > "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line > 1298, in configure_renewal > self.dogtag_constants.ALIAS_DIR, 'renew_ca_cert "%s"' % nickname) > > File "/usr/lib/python2.7/site-packages/ipapython/certmonger.py", line > 394, in dogtag_start_tracking > (stdout, stderr, returncode) = ipautil.run(args, nolog=[pin]) > > File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line > 309, in run > raise CalledProcessError(p.returncode, args) > > 2012-09-11T18:28:52Z INFO The ipa-upgradeconfig command failed, > exception: CalledProcessError: Command '/usr/bin/getcert start-tracking > -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c > dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert > "auditSigningCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit status 1 > > I'm not sure if this is related to this patch or not. If it isn't can > you file a new ticket on it, or add it the 2.2 -> 3.0 upgrade ticket? These are both easily reproducable. Rather than restarting the world this time I looked to see what was running: # ipactl status Directory Service: RUNNING KDC Service: RUNNING KPASSWD Service: RUNNING MEMCACHE Service: RUNNING HTTP Service: RUNNING CA Service: RUNNING # ps -ef|grep java # /bin/systemctl -a status pki-cad at pki-ca.service pki-cad at pki-ca.service - PKI Certificate Authority Server pki-ca Loaded: loaded (/usr/lib/systemd/system/pki-cad at .service; enabled) Active: failed (Result: exit-code) since Tue, 11 Sep 2012 15:10:35 -0400; 8min ago Process: 5917 ExecStop=/usr/bin/pkicontrol stop ca %i (code=exited, status=1/FAILURE) Process: 5763 ExecStart=/usr/bin/pkicontrol start ca %i (code=exited, status=0/SUCCESS) Main PID: 5830 (code=exited, status=0/SUCCESS) CGroup: name=systemd:/system/pki-cad at .service/pki-ca Sep 11 15:10:27 edsel.greyoak.com runuser[5792]: pam_unix(runuser-l:session): session opened for user pkiuser by (uid=0) Sep 11 15:10:27 edsel.greyoak.com runuser[5792]: pam_unix(runuser-l:session): session closed for user pkiuser Sep 11 15:10:35 edsel.greyoak.com runuser[5936]: pam_unix(runuser-l:session): session opened for user pkiuser by (uid=0) Sep 11 15:10:35 edsel.greyoak.com runuser[5936]: pam_unix(runuser-l:session): session closed for user pkiuser So I'm thinking that IPA believes that the CA is running, and systemd reported it started, but something went wrong. The catalina.out log doesn't really show a failure, just the stupid "I can't stop because I'm already stopped" traceback. I wonder if this is an issue with pkicontrol where restart doesn't handle a stop failure, but I'm just guessing. I suspect that the start-tracking issue is not related. rob From rcritten at redhat.com Tue Sep 11 19:22:51 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 11 Sep 2012 15:22:51 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <504F86C0.6060907@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502B86F5.4070008@redhat.com> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> <1347329065.24731.101.camel@aleeredhat.laptop> <504F318F.3030603@redhat.com> <504F35AD.3030905@redhat.com> <1347373691.24731.125.camel@aleeredhat.laptop> <504F4CE0.9080706@redhat.com> <504F67C5.4000600@redhat.com> <504F86C0.6060907@redhat.com> Message-ID: <504F8F8B.7040709@redhat.com> Rob Crittenden wrote: > Petr Viktorin wrote: >> On 09/11/2012 04:38 PM, Rob Crittenden wrote: >>> Ade Lee wrote: >>>> On Tue, 2012-09-11 at 08:59 -0400, Rob Crittenden wrote: >>>>> Petr Viktorin wrote: >>>>>> On 09/11/2012 04:04 AM, Ade Lee wrote: >>>>>>> On Mon, 2012-09-10 at 16:58 -0400, Rob Crittenden wrote: >>>>>>>> Petr Viktorin wrote: >>>>>>>>> Attaching rebased and squashed patches. I've done some testing >>>>>>>>> with >>>>>>>>> them >>>>>>>>> but please test some more. >>>>>>>>> >>>>>>>> >>>>>>>> Most of these aren't IPA issues, but dogtag issues. I'll try to >>>>>>>> split >>>>>>>> them out. >>>>>>>> >>>>>>>> IPA: >>>>>>>> >>>>>>>> For the configuration files in install/conf to be updated at rpm >>>>>>>> update >>>>>>>> time the VERSION needs to be incremented. >>>>>> >>>>>> These files should stay the same since on upgrade we're still using a >>>>>> Dogtag 9 style instance. The Dogtag 10 ports are only used in new >>>>>> installs. >>>>>> >>>>>>>> The ipa package lacks any updated dogtag dependencies, so I abused >>>>>>>> it. >>>>>> >>>>>> What should the updated dependencies be? Since it should work with >>>>>> both >>>>>> dogtag 9 and 10, I don't see how they should change. >>>>> >>>>> I don't know either, but we need to prevent people from installing >>>>> incompatible package combinations. >>>>> >>>> Would'nt the Conflicts: ipa < 3.0 in pki-ca mentioned below satisfy >>>> this >>>> requirement? The main concern is that you must have ipa 3.0 if you >>>> have >>>> dogtag 10. >>>> >>>> Given that dogtag is consumed by IPA though, it makes more sense to put >>>> the relevant conflicts in IPA rather than in dogtag. So in this case, >>>> that would mean putting Conflicts: pki-ca >= 10.0 in IPA 2.x. >>>> Recall that dogtag 10 will only be officially available in f18+. >>> >>> That isn't enough. If a F-17 user with IPA 2.2 installed upgrades to >>> F-18 they would be able to install dogtag 10 and blow up their IPA >>> server. >>> >>>> >>>>>>>> I installed IPA with dogtag 9 and created a replica. >>>>>>>> >>>>>>>> I updated the IPA bits, that worked fine. >>>>>>>> >>>>>>>> I updated to dogtag 10 and now the CA doesn't work on the master, >>>>>>>> including starting the dogtag instance. Note that the rpm update >>>>>>>> process >>>>>>>> worked, no notice that the CA service didn't restart. >>>>>>>> >>>>>>> Did you try to restart the CA with selinux in permissive mode? >>>>>>> This is >>>>>>> still required right now until I get the selinux policy all >>>>>>> straightened >>>>>>> out. >>>>>>> >>>>>>> There is also a separate dogtag ticket (which is currently being >>>>>>> worked >>>>>>> on) to restart existing dogtag instances when dogtag is upgraded >>>>>>> from >>>>>>> 9->10. >>>>>> >>>>>> In permissive mode, this upgrade works for me. >>>>> >>>>> I was in enforcing mode but saw no AVCs. What is the ETA on fixing >>>>> this? >>>>> >>>> >>>> Within the next week or two, I need to finish the IPA merge database >>>> patch first, and then co-ordinate changes with the selinux guys. >>>> >>>>>> >>>>>> >>>>>> Sometimes I do get this error intermittently: >>>>>> >>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to >>>>>> communicate with CMS (Service Temporarily Unavailable) >>>>>> >>>>>> Usually, waiting a couple of minutes clears this up. Perhaps we are >>>>>> not >>>>>> doing startup detection correctly. Ade mentioned that waiting for >>>>>> ports >>>>>> may not be ideal. How do we know if Dogtag is initialized? >>>>> >>>>> Years ago we had discussed with Andrew and Matt creating a URI that >>>>> can >>>>> be queried to determine dogtag status. I don't think that ever went >>>>> anywhere. >>>>> >>>> Petr, this happens only on reboot, right? And not on regular "service >>>> ipa restart"? >> >> I've now seen it happen right after a 9 ? 10 upgrade. >> >>>> Yeah, I remember this conversation - and even created a bug for it at >>>> some point. This went away because the mechanism you were using seemed >>>> to be working. The timing may be very different now with tomcat 7 and >>>> systemd. I'll open a dogtag trac ticket for this. >>> >>> Ok. >>> >>>> >>>>>> >>>>>>>> Uninstalling failed because it tried to run pkidestroy and not >>>>>>>> pkiremove. >>>>>> >>>>>> I was under the impression that pkidestroy was the correct command to >>>>>> remove an upgraded instance. I'll check with Ade. >>>>>> >>>>>>> I'll test this too. >>>>>>> >>>>>>>> The contents of the file passed to pkispawn should be logged so we >>>>>>>> can >>>>>>>> see exactly what was passed in. >>>>>>>> >>>>>>> Its a pretty big file. You might want to only log the >>>>>>> modifications. >>>>>>> Or save the file somewhere. >>>>>> >>>>>> Our logs are pretty verbose, so that shouldn't be a problem. I'll >>>>>> put it >>>>>> in the next version of the patch. >>>>> >>>>> The question to ask is: would you need the contents of this file if >>>>> all >>>>> you got were logs and needed to evaluate why installation failed? In >>>>> most cases this is yes. >>>>> >>>> Up to you guys. There is a patch I am working on in which I will be >>>> logging the object that is being passed to the server from pkispawn. >>>> That - and the diffs to the standard config file as I mentioned above - >>>> will likely be sufficient to debug almost all cases. >>>> >>>> Make sure not to log any passwords. >>>> >> >> Thanks for the catch. Attaching updated patch that sanitizes the >> passwords. >> >>>>>>>> DOGTAG: >>>>>>>> >>>>>>>> When upgrading using the dogtag-devel repo I had to specify >>>>>>>> pki-tools.x86_64 otherwise it tried to install both 32 and 64-bit >>>>>>>> versions (and failed). >>>>>>>> >>>>>>>> I ended up running: yum update pki-ca tomcatjss pki-tools.x86_64 >>>>>>>> --enablerepo=dogtag-devel --enablerepo=updates-testing >>>>>>>> >>>>>>> We'll look into this. I think I know why this happens. >>>>>>> >>>>>>>> What happens if someone manually upgrades pki-ca without first >>>>>>>> updating >>>>>>>> ipa? I think that pki-ca is going to need a Conflicts ipa < 3.0 in >>>>>>>> it. >>>>>>> >>>>>>> We can add that. >>>>>>> >>>>>>>> certificate renewal failed. I spent far too long trying to figure >>>>>>>> out >>>>>>>> why tomcat wasn't listening on port 9180 but failed. I think >>>>>>>> 9180 is >>>>>>>> actually the old server, right? So another missing dependency on a >>>>>>>> fixed >>>>>>>> certmonger? >>>>>>>> >>>>>>>> The best I could find was the certmonger error: >>>>>>>> >>>>>>>> ca-error: Error 7 connecting to >>>>>>>> http://edsel.example.com:9180/ca/ee/ca/profileSubmit: Couldn't >>>>>>>> connect >>>>>>>> to server. >>>>>>>> >>>>>> >>>>>> I'll set the certmonger min version to 0.60 as per Nalin's mail. >>>>> >>>>> Ok. >>>>> >>>>>>> Is this cert renewal on a dogtag 10 instance? Or the upgraded >>>>>>> dogtag 9? >>>>>>> If its dogtag 10, perhaps you do not have the certmonger version >>>>>>> that >>>>>>> has the relevant fix? If its dogtag 9, then we need to figure out >>>>>>> whats >>>>>>> going on. That reminds me - I need to file a bug to allow >>>>>>> certmonger to >>>>>>> talk to the newly defined dogtag ports. Do you have selinux >>>>>>> permissive? >>>>>>> >>>>>>>> There is no man page for pkispawn/pkidestroy :-( According to the >>>>>>>> FHS >>>>>>>> these should not be in /bin but in /usr/sbin (not end-user >>>>>>>> commands). >>>>>>>> >>>>>>> There is a trac ticket open for man pages for pkispawn and >>>>>>> pkidestroy. >>>>>>> We plan to complete this ticket by the time f18 is released. >>>>>>> >>>>>>> We'll look into the location of pkispawn/pkicreate. >>>>>>> >>>>>>>> The output of pkicreate/pkisilent was really terrible and not >>>>>>>> usable at >>>>>>>> all so we didn't display it when failures occurred. It looks like >>>>>>>> that >>>>>>>> has been addressed, at least for the case where a CA is already >>>>>>>> configured and you try to install IPA. Perhaps we should capture >>>>>>>> stderr >>>>>>>> and display that instead of the command-line of pkispawn? Again, a >>>>>>>> man >>>>>>>> page would help with the integration. >>>>>>>> >>>>>>>> 2012-09-10T20:51:45Z DEBUG [2/18]: configuring certificate server >>>>>>>> instance >>>>>>>> 2012-09-10T20:51:45Z DEBUG args=/bin/pkispawn -s CA -f >>>>>>>> /tmp/tmp_Urraq >>>>>>>> 2012-09-10T20:51:45Z DEBUG stdout= >>>>>>>> 2012-09-10T20:51:45Z DEBUG stderr=pkispawn : ERROR ....... >>>>>>>> PKI >>>>>>>> subsystem 'CA' for instance 'pki-tomcat' already exists! >>>>>>>> >>>>>>>> 2012-09-10T20:51:45Z CRITICAL failed to configure ca instance >>>>>>>> Command >>>>>>>> '/bin/pkispawn -s CA -f /tmp/tmp_Urraq' returned non-zero exit >>>>>>>> status 1 >>>>>>>> >>>>>>> That may be a good idea. Of course. thats an IPA thing, right? >>>>> >>>>> Right, not a show-stopper for this but a new ticket should be >>>>> opened to >>>>> track it. >> >> https://fedorahosted.org/freeipa/ticket/3072 > > Thanks. > > I tested a 2.2 -> 3.0 upgrade with dogtag 9 and it seems to have failed > to restart something: > > [ post rpm -Uvh dist/rpms/freeipa*.rpm ] > > [root at edsel freeipa]# ipa cert-show 1 > ipa: ERROR: Certificate operation cannot be completed: Unable to > communicate with CMS (Service Temporarily Unavailable) > [root at edsel freeipa]# ipactl restart > Restarting Directory Service > Restarting KDC Service > Restarting KPASSWD Service > Restarting MEMCACHE Service > Restarting HTTP Service > Restarting CA Service > [root at edsel freeipa]# ipa cert-show 1 > Certificate: MIIDjTCCAnWgAwIBAgIBATANBgk > ... > > The apache log had this: > > [Tue Sep 11 14:35:42 2012] [error] (111)Connection refused: proxy: AJP: > attempt to connect to 127.0.0.1:9447 (localhost) failed > [Tue Sep 11 14:35:42 2012] [error] ap_proxy_connect_backend disabling > worker for (localhost) > [Tue Sep 11 14:35:42 2012] [error] proxy: AJP: failed to make connection > to backend: localhost > > So I'm gathering that dogtag didn't restart properly, but I'm just > guessing. It could have been the PKI-IPA ds instance too, I'm not sure > where to check. > > I also noticed this in /var/log/ipaupgrade.log: > > 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl start certmonger.service > 2012-09-11T18:28:22Z DEBUG stdout= > 2012-09-11T18:28:22Z DEBUG stderr= > 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl is-active certmonger.service > 2012-09-11T18:28:22Z DEBUG stdout=active > ... > ... [ snip certutil output ] > ... > 2012-09-11T18:28:52Z DEBUG args=/usr/bin/getcert start-tracking -d > /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c > dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert > "auditSigningCert cert-pki-ca" -P XXXXXXXX > 2012-09-11T18:28:52Z DEBUG stdout=Please verify that the certmonger > service is still running. > > 2012-09-11T18:28:52Z DEBUG stderr= > 2012-09-11T18:28:52Z INFO File > "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", > line 614, in run_script > return_value = main_function() > > File "/usr/sbin/ipa-upgradeconfig", line 540, in main > enable_certificate_renewal(api.env.realm) > > File "/usr/sbin/ipa-upgradeconfig", line 455, in > enable_certificate_renewal > ca.configure_renewal() > > File > "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line > 1298, in configure_renewal > self.dogtag_constants.ALIAS_DIR, 'renew_ca_cert "%s"' % nickname) > > File "/usr/lib/python2.7/site-packages/ipapython/certmonger.py", line > 394, in dogtag_start_tracking > (stdout, stderr, returncode) = ipautil.run(args, nolog=[pin]) > > File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line > 309, in run > raise CalledProcessError(p.returncode, args) > > 2012-09-11T18:28:52Z INFO The ipa-upgradeconfig command failed, > exception: CalledProcessError: Command '/usr/bin/getcert start-tracking > -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c > dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert > "auditSigningCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit status 1 > > I'm not sure if this is related to this patch or not. If it isn't can > you file a new ticket on it, or add it the 2.2 -> 3.0 upgrade ticket? > > rob And to reply to myself, how do we imagine that upgrades will work? Is it legal for someone to upgrade to IPA 3.0 and dogtag 10 separately, or do we expect/require it be done at the same time, or one first? rob From rcritten at redhat.com Tue Sep 11 19:38:27 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 11 Sep 2012 15:38:27 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <504F8F8B.7040709@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> <1347329065.24731.101.camel@aleeredhat.laptop> <504F318F.3030603@redhat.com> <504F35AD.3030905@redhat.com> <1347373691.24731.125.camel@aleeredhat.laptop> <504F4CE0.9080706@redhat.com> <504F67C5.4000600@redhat.com> <504F86C0.6060907@redhat.com> <504F8F8B.7040709@redhat.com> Message-ID: <504F9333.8040707@redhat.com> Rob Crittenden wrote: > Rob Crittenden wrote: >> Petr Viktorin wrote: >>> On 09/11/2012 04:38 PM, Rob Crittenden wrote: >>>> Ade Lee wrote: >>>>> On Tue, 2012-09-11 at 08:59 -0400, Rob Crittenden wrote: >>>>>> Petr Viktorin wrote: >>>>>>> On 09/11/2012 04:04 AM, Ade Lee wrote: >>>>>>>> On Mon, 2012-09-10 at 16:58 -0400, Rob Crittenden wrote: >>>>>>>>> Petr Viktorin wrote: >>>>>>>>>> Attaching rebased and squashed patches. I've done some testing >>>>>>>>>> with >>>>>>>>>> them >>>>>>>>>> but please test some more. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Most of these aren't IPA issues, but dogtag issues. I'll try to >>>>>>>>> split >>>>>>>>> them out. >>>>>>>>> >>>>>>>>> IPA: >>>>>>>>> >>>>>>>>> For the configuration files in install/conf to be updated at rpm >>>>>>>>> update >>>>>>>>> time the VERSION needs to be incremented. >>>>>>> >>>>>>> These files should stay the same since on upgrade we're still >>>>>>> using a >>>>>>> Dogtag 9 style instance. The Dogtag 10 ports are only used in new >>>>>>> installs. >>>>>>> >>>>>>>>> The ipa package lacks any updated dogtag dependencies, so I abused >>>>>>>>> it. >>>>>>> >>>>>>> What should the updated dependencies be? Since it should work with >>>>>>> both >>>>>>> dogtag 9 and 10, I don't see how they should change. >>>>>> >>>>>> I don't know either, but we need to prevent people from installing >>>>>> incompatible package combinations. >>>>>> >>>>> Would'nt the Conflicts: ipa < 3.0 in pki-ca mentioned below satisfy >>>>> this >>>>> requirement? The main concern is that you must have ipa 3.0 if you >>>>> have >>>>> dogtag 10. >>>>> >>>>> Given that dogtag is consumed by IPA though, it makes more sense to >>>>> put >>>>> the relevant conflicts in IPA rather than in dogtag. So in this case, >>>>> that would mean putting Conflicts: pki-ca >= 10.0 in IPA 2.x. >>>>> Recall that dogtag 10 will only be officially available in f18+. >>>> >>>> That isn't enough. If a F-17 user with IPA 2.2 installed upgrades to >>>> F-18 they would be able to install dogtag 10 and blow up their IPA >>>> server. >>>> >>>>> >>>>>>>>> I installed IPA with dogtag 9 and created a replica. >>>>>>>>> >>>>>>>>> I updated the IPA bits, that worked fine. >>>>>>>>> >>>>>>>>> I updated to dogtag 10 and now the CA doesn't work on the master, >>>>>>>>> including starting the dogtag instance. Note that the rpm update >>>>>>>>> process >>>>>>>>> worked, no notice that the CA service didn't restart. >>>>>>>>> >>>>>>>> Did you try to restart the CA with selinux in permissive mode? >>>>>>>> This is >>>>>>>> still required right now until I get the selinux policy all >>>>>>>> straightened >>>>>>>> out. >>>>>>>> >>>>>>>> There is also a separate dogtag ticket (which is currently being >>>>>>>> worked >>>>>>>> on) to restart existing dogtag instances when dogtag is upgraded >>>>>>>> from >>>>>>>> 9->10. >>>>>>> >>>>>>> In permissive mode, this upgrade works for me. >>>>>> >>>>>> I was in enforcing mode but saw no AVCs. What is the ETA on fixing >>>>>> this? >>>>>> >>>>> >>>>> Within the next week or two, I need to finish the IPA merge database >>>>> patch first, and then co-ordinate changes with the selinux guys. >>>>> >>>>>>> >>>>>>> >>>>>>> Sometimes I do get this error intermittently: >>>>>>> >>>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to >>>>>>> communicate with CMS (Service Temporarily Unavailable) >>>>>>> >>>>>>> Usually, waiting a couple of minutes clears this up. Perhaps we are >>>>>>> not >>>>>>> doing startup detection correctly. Ade mentioned that waiting for >>>>>>> ports >>>>>>> may not be ideal. How do we know if Dogtag is initialized? >>>>>> >>>>>> Years ago we had discussed with Andrew and Matt creating a URI that >>>>>> can >>>>>> be queried to determine dogtag status. I don't think that ever went >>>>>> anywhere. >>>>>> >>>>> Petr, this happens only on reboot, right? And not on regular "service >>>>> ipa restart"? >>> >>> I've now seen it happen right after a 9 ? 10 upgrade. >>> >>>>> Yeah, I remember this conversation - and even created a bug for it at >>>>> some point. This went away because the mechanism you were using >>>>> seemed >>>>> to be working. The timing may be very different now with tomcat 7 and >>>>> systemd. I'll open a dogtag trac ticket for this. >>>> >>>> Ok. >>>> >>>>> >>>>>>> >>>>>>>>> Uninstalling failed because it tried to run pkidestroy and not >>>>>>>>> pkiremove. >>>>>>> >>>>>>> I was under the impression that pkidestroy was the correct >>>>>>> command to >>>>>>> remove an upgraded instance. I'll check with Ade. >>>>>>> >>>>>>>> I'll test this too. >>>>>>>> >>>>>>>>> The contents of the file passed to pkispawn should be logged so we >>>>>>>>> can >>>>>>>>> see exactly what was passed in. >>>>>>>>> >>>>>>>> Its a pretty big file. You might want to only log the >>>>>>>> modifications. >>>>>>>> Or save the file somewhere. >>>>>>> >>>>>>> Our logs are pretty verbose, so that shouldn't be a problem. I'll >>>>>>> put it >>>>>>> in the next version of the patch. >>>>>> >>>>>> The question to ask is: would you need the contents of this file if >>>>>> all >>>>>> you got were logs and needed to evaluate why installation failed? In >>>>>> most cases this is yes. >>>>>> >>>>> Up to you guys. There is a patch I am working on in which I will be >>>>> logging the object that is being passed to the server from pkispawn. >>>>> That - and the diffs to the standard config file as I mentioned >>>>> above - >>>>> will likely be sufficient to debug almost all cases. >>>>> >>>>> Make sure not to log any passwords. >>>>> >>> >>> Thanks for the catch. Attaching updated patch that sanitizes the >>> passwords. >>> >>>>>>>>> DOGTAG: >>>>>>>>> >>>>>>>>> When upgrading using the dogtag-devel repo I had to specify >>>>>>>>> pki-tools.x86_64 otherwise it tried to install both 32 and 64-bit >>>>>>>>> versions (and failed). >>>>>>>>> >>>>>>>>> I ended up running: yum update pki-ca tomcatjss pki-tools.x86_64 >>>>>>>>> --enablerepo=dogtag-devel --enablerepo=updates-testing >>>>>>>>> >>>>>>>> We'll look into this. I think I know why this happens. >>>>>>>> >>>>>>>>> What happens if someone manually upgrades pki-ca without first >>>>>>>>> updating >>>>>>>>> ipa? I think that pki-ca is going to need a Conflicts ipa < 3.0 in >>>>>>>>> it. >>>>>>>> >>>>>>>> We can add that. >>>>>>>> >>>>>>>>> certificate renewal failed. I spent far too long trying to figure >>>>>>>>> out >>>>>>>>> why tomcat wasn't listening on port 9180 but failed. I think >>>>>>>>> 9180 is >>>>>>>>> actually the old server, right? So another missing dependency on a >>>>>>>>> fixed >>>>>>>>> certmonger? >>>>>>>>> >>>>>>>>> The best I could find was the certmonger error: >>>>>>>>> >>>>>>>>> ca-error: Error 7 connecting to >>>>>>>>> http://edsel.example.com:9180/ca/ee/ca/profileSubmit: Couldn't >>>>>>>>> connect >>>>>>>>> to server. >>>>>>>>> >>>>>>> >>>>>>> I'll set the certmonger min version to 0.60 as per Nalin's mail. >>>>>> >>>>>> Ok. >>>>>> >>>>>>>> Is this cert renewal on a dogtag 10 instance? Or the upgraded >>>>>>>> dogtag 9? >>>>>>>> If its dogtag 10, perhaps you do not have the certmonger version >>>>>>>> that >>>>>>>> has the relevant fix? If its dogtag 9, then we need to figure out >>>>>>>> whats >>>>>>>> going on. That reminds me - I need to file a bug to allow >>>>>>>> certmonger to >>>>>>>> talk to the newly defined dogtag ports. Do you have selinux >>>>>>>> permissive? >>>>>>>> >>>>>>>>> There is no man page for pkispawn/pkidestroy :-( According to the >>>>>>>>> FHS >>>>>>>>> these should not be in /bin but in /usr/sbin (not end-user >>>>>>>>> commands). >>>>>>>>> >>>>>>>> There is a trac ticket open for man pages for pkispawn and >>>>>>>> pkidestroy. >>>>>>>> We plan to complete this ticket by the time f18 is released. >>>>>>>> >>>>>>>> We'll look into the location of pkispawn/pkicreate. >>>>>>>> >>>>>>>>> The output of pkicreate/pkisilent was really terrible and not >>>>>>>>> usable at >>>>>>>>> all so we didn't display it when failures occurred. It looks like >>>>>>>>> that >>>>>>>>> has been addressed, at least for the case where a CA is already >>>>>>>>> configured and you try to install IPA. Perhaps we should capture >>>>>>>>> stderr >>>>>>>>> and display that instead of the command-line of pkispawn? Again, a >>>>>>>>> man >>>>>>>>> page would help with the integration. >>>>>>>>> >>>>>>>>> 2012-09-10T20:51:45Z DEBUG [2/18]: configuring certificate >>>>>>>>> server >>>>>>>>> instance >>>>>>>>> 2012-09-10T20:51:45Z DEBUG args=/bin/pkispawn -s CA -f >>>>>>>>> /tmp/tmp_Urraq >>>>>>>>> 2012-09-10T20:51:45Z DEBUG stdout= >>>>>>>>> 2012-09-10T20:51:45Z DEBUG stderr=pkispawn : ERROR ....... >>>>>>>>> PKI >>>>>>>>> subsystem 'CA' for instance 'pki-tomcat' already exists! >>>>>>>>> >>>>>>>>> 2012-09-10T20:51:45Z CRITICAL failed to configure ca instance >>>>>>>>> Command >>>>>>>>> '/bin/pkispawn -s CA -f /tmp/tmp_Urraq' returned non-zero exit >>>>>>>>> status 1 >>>>>>>>> >>>>>>>> That may be a good idea. Of course. thats an IPA thing, right? >>>>>> >>>>>> Right, not a show-stopper for this but a new ticket should be >>>>>> opened to >>>>>> track it. >>> >>> https://fedorahosted.org/freeipa/ticket/3072 >> >> Thanks. >> >> I tested a 2.2 -> 3.0 upgrade with dogtag 9 and it seems to have failed >> to restart something: >> >> [ post rpm -Uvh dist/rpms/freeipa*.rpm ] >> >> [root at edsel freeipa]# ipa cert-show 1 >> ipa: ERROR: Certificate operation cannot be completed: Unable to >> communicate with CMS (Service Temporarily Unavailable) >> [root at edsel freeipa]# ipactl restart >> Restarting Directory Service >> Restarting KDC Service >> Restarting KPASSWD Service >> Restarting MEMCACHE Service >> Restarting HTTP Service >> Restarting CA Service >> [root at edsel freeipa]# ipa cert-show 1 >> Certificate: MIIDjTCCAnWgAwIBAgIBATANBgk >> ... >> >> The apache log had this: >> >> [Tue Sep 11 14:35:42 2012] [error] (111)Connection refused: proxy: AJP: >> attempt to connect to 127.0.0.1:9447 (localhost) failed >> [Tue Sep 11 14:35:42 2012] [error] ap_proxy_connect_backend disabling >> worker for (localhost) >> [Tue Sep 11 14:35:42 2012] [error] proxy: AJP: failed to make connection >> to backend: localhost >> >> So I'm gathering that dogtag didn't restart properly, but I'm just >> guessing. It could have been the PKI-IPA ds instance too, I'm not sure >> where to check. >> >> I also noticed this in /var/log/ipaupgrade.log: >> >> 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl start certmonger.service >> 2012-09-11T18:28:22Z DEBUG stdout= >> 2012-09-11T18:28:22Z DEBUG stderr= >> 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl is-active >> certmonger.service >> 2012-09-11T18:28:22Z DEBUG stdout=active >> ... >> ... [ snip certutil output ] >> ... >> 2012-09-11T18:28:52Z DEBUG args=/usr/bin/getcert start-tracking -d >> /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c >> dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert >> "auditSigningCert cert-pki-ca" -P XXXXXXXX >> 2012-09-11T18:28:52Z DEBUG stdout=Please verify that the certmonger >> service is still running. >> >> 2012-09-11T18:28:52Z DEBUG stderr= >> 2012-09-11T18:28:52Z INFO File >> "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", >> line 614, in run_script >> return_value = main_function() >> >> File "/usr/sbin/ipa-upgradeconfig", line 540, in main >> enable_certificate_renewal(api.env.realm) >> >> File "/usr/sbin/ipa-upgradeconfig", line 455, in >> enable_certificate_renewal >> ca.configure_renewal() >> >> File >> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line >> 1298, in configure_renewal >> self.dogtag_constants.ALIAS_DIR, 'renew_ca_cert "%s"' % nickname) >> >> File "/usr/lib/python2.7/site-packages/ipapython/certmonger.py", line >> 394, in dogtag_start_tracking >> (stdout, stderr, returncode) = ipautil.run(args, nolog=[pin]) >> >> File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line >> 309, in run >> raise CalledProcessError(p.returncode, args) >> >> 2012-09-11T18:28:52Z INFO The ipa-upgradeconfig command failed, >> exception: CalledProcessError: Command '/usr/bin/getcert start-tracking >> -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c >> dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert >> "auditSigningCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit >> status 1 >> >> I'm not sure if this is related to this patch or not. If it isn't can >> you file a new ticket on it, or add it the 2.2 -> 3.0 upgrade ticket? >> >> rob > > And to reply to myself, how do we imagine that upgrades will work? > > Is it legal for someone to upgrade to IPA 3.0 and dogtag 10 separately, > or do we expect/require it be done at the same time, or one first? > Sorry to break this into a ton of e-mails. I updated pki-ca and getting a selinux error in permissive: # /bin/systemctl -a status pki-cad at pki-ca.service pki-cad at pki-ca.service - PKI Certificate Authority Server pki-ca Loaded: loaded (/usr/lib/systemd/system/pki-cad at .service; enabled) Active: failed (Result: exit-code) since Tue, 11 Sep 2012 15:36:37 -0400; 3s ago Process: 7067 ExecStart=/usr/bin/pkicontrol start ca %i (code=exited, status=1/FAILURE) Main PID: 5830 (code=exited, status=0/SUCCESS) CGroup: name=systemd:/system/pki-cad at .service/pki-ca Sep 11 15:36:37 edsel.greyoak.com pkicontrol[7067]: /usr/bin/runcon: invalid context: system_u:system_r:pki_ca_script_t:s0: Invalid argument I assume this means I need to re-install pki-selinux? It should be possible to tweak the dependencies such that this package is installed first. rob From rmeggins at redhat.com Tue Sep 11 20:01:29 2012 From: rmeggins at redhat.com (Rich Megginson) Date: Tue, 11 Sep 2012 14:01:29 -0600 Subject: [Freeipa-devel] Ticket #2866 - referential integrity in IPA In-Reply-To: <504F6C16.30400@redhat.com> References: <1345211089.3796.34.camel@priserak> <502E4E78.1090407@redhat.com> <503B9F4E.4010005@redhat.com> <503BA065.1060207@redhat.com> <5044C1F0.7080609@redhat.com> <504612DC.9060402@redhat.com> <504F6C16.30400@redhat.com> Message-ID: <504F9899.8010401@redhat.com> On 09/11/2012 10:51 AM, Martin Kosek wrote: > On 09/04/2012 04:40 PM, Rich Megginson wrote: >> On 09/03/2012 08:42 AM, Martin Kosek wrote: >>> On 08/27/2012 06:29 PM, Rich Megginson wrote: > ... >>> This is the plan I plan to take: >>> 1) Add pres,eq indexes for all un-indexed attributes that we want to check: >>> sourcehost >>> memberservice >>> managedby >>> memberallowcmd >>> memberdenycmd >>> ipasudorunas >>> ipasudorunasgroup >> ok > ... > > Implementation of the Referential Integrity in IPA works OK so far, I just hit > a strange issue when indexing memberallowcmd and memberdenycmd attributes in IPA: > > dirsrv errors log: > ... > [11/Sep/2012:11:39:53 -0400] - The attribute [memberdenycmd] does not have a > valid ORDERING matching rule - error 2:s > [11/Sep/2012:11:39:58 -0400] - userRoot: Indexing attribute: memberdenycmd > [11/Sep/2012:11:39:58 -0400] - userRoot: Finished indexing. > ... > > This cause RI to fail to handle this attribute in IPA (which is expected based > on the error message). > > I checked the attributes types, they look like that: > > attributetypes: ( 2.16.840.1.113730.3.8.7.1 NAME 'memberAllowCmd' DESC 'Refere > nce to a command or group of commands that are allowed by the rule.' SUP dist > inguishedName EQUALITY distinguishedNameMatch ORDERING distinguishedNameMatch > SUBSTR distinguishedNameMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.12 X-ORIGIN > 'IPA v2' ) > > attributetypes: ( 2.16.840.1.113730.3.8.7.2 NAME 'memberDenyCmd' DESC 'Referen > ce to a command or group of commands that are denied by the rule.' SUP distin > guishedName EQUALITY distinguishedNameMatch ORDERING distinguishedNameMatch S > UBSTR distinguishedNameMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.12 X-ORIGIN 'I > PA v2' ) > > "distinguishedNameMatch" ORDERING rule looks wrong, it is only a matching rule. > Anyone knows why it was defined that way? Or what should be a correct setting > for this attribute instead? caseIgnoreOrderingMatch seems as a logical choice... Not sure. The problem is that, in LDAP, there isn't a concept of "ordering" or "substring" matching on DN values. http://www.ietf.org/rfc/rfc4517.txt - there is only distinguishedNameMatch which is an EQUALITY rule. Do you really need ordering and substring here? The problem with using some other ordering or substring matching rule is that it will not properly normalize the DN valued string, so you may get incorrect results. > > I will have to fix the attributeTypes definition in IPA before we can enable > index& RI for them. > > Thanks, > Martin From rcritten at redhat.com Tue Sep 11 20:39:46 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 11 Sep 2012 16:39:46 -0400 Subject: [Freeipa-devel] [PATCH] 0078 ipa-client-install: Obtain host TGT from one specific KDC In-Reply-To: <5048929E.7080605@redhat.com> References: <5048929E.7080605@redhat.com> Message-ID: <504FA192.5070000@redhat.com> Petr Viktorin wrote: > When installing the client, we need to take extra case to only contact > the one server we're installing against. Otherwise, in the real world, > we might hit a server that hasn't replicated info about the client yet. > > This patch fixes a bug where kinit attempted to contact a KDC that > didn't have the host principal yet. > > > To reproduce: > > - Install a "master" and "replica" > - Change the Kerberos DNS entries to only point to the replica: > for REC_NAME in '_kerberos-master._tcp' '_kerberos-master._udp' > '_kerberos._tcp' '_kerberos._udp' '_kpasswd._tcp' '_kpasswd._udp'; do > ipa dnsrecord-mod $DOMAIN $REC_NAME --srv-rec="0 100 88 > $REPLICA_HOSTNAME" > done > ipa dnsrecord-mod $DOMAIN _ldap._tcp --srv-rec="0 100 389 > $MASTER_HOSTNAME" > ipa dnsrecord-find $DOMAIN # check > - Sever communication between the hosts to disable replication: > (on master) > iptables -A INPUT -j DROP -p all --source $REPLICA_IP > - On client machine, put master as nameserver in /etc/resolv.conf & > install client > > This will fail without the patch. > > > Thanks to Petr Spacek, Simo, and Scott for helping to reproduce and > explain the bug. I learned a lot. > > https://fedorahosted.org/freeipa/ticket/2982 ACK, pushed to master and ipa-3-0 rob From rcritten at redhat.com Tue Sep 11 21:05:39 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 11 Sep 2012 17:05:39 -0400 Subject: [Freeipa-devel] [PATCH] 0077 Check direct/reverse hostname/address resolution in ipa-replica-install In-Reply-To: <5046FF0E.1000504@redhat.com> References: <5046009E.2020306@redhat.com> <50463DE6.9070300@redhat.com> <5046FF0E.1000504@redhat.com> Message-ID: <504FA7A3.5080201@redhat.com> Petr Viktorin wrote: > On 09/04/2012 07:44 PM, Rob Crittenden wrote: >> Petr Viktorin wrote: >>> >>> https://fedorahosted.org/freeipa/ticket/2845 >> >> Shouldn't this also call verify_fqdn() on the local hostname and not >> just the master? I think this would eventually fail in the conncheck but >> what if that was skipped? >> >> rob > > A few lines above there is a call to get_host_name, which will call > verify_fqdn. > I double-checked this, it fails in conncheck. Here are my steps: # ipa-server-install --setup-dns # ipa-replica-prepare replica.example.com --ip-address=192.168.100.2 # ipa host-del replica.example.com On replica, set DNS to IPA master, with hostname in /etc/hosts. # ipa-replica-install ... The verify_fqdn() passes because the resolver uses /etc/hosts. The conncheck fails: Execute check on remote master Check connection from master to remote replica 'replica.example.com': Remote master check failed with following error message(s): Could not chdir to home directory /home/admin: No such file or directory Port check failed! Unable to resolve host name 'replica.example.com' Connection check failed! Please fix your network settings according to error messages above. If the check results are not valid it can be skipped with --skip-conncheck parameter. The DNS test happens much further after this, and I get why, I just don't see how useful it is unless the --skip-conncheck is used. ipa-replica-install ~rcrit/replica-info-replica.example.com.gpg --skip-conncheck Directory Manager (existing master) password: ipa : ERROR Could not resolve hostname replica.example.com using DNS. Clients may not function properly. Please check your DNS setup. (Note that this check queries IPA DNS directly and ignores /etc/hosts.) Continue? [no]: So I guess, what are the intentions here? It is certainly better than before. rob From sbingram at gmail.com Wed Sep 12 00:59:47 2012 From: sbingram at gmail.com (Stephen Ingram) Date: Tue, 11 Sep 2012 17:59:47 -0700 Subject: [Freeipa-devel] slow response In-Reply-To: <1344372819.20530.314.camel@willson.li.ssimo.org> References: <50070E55.90307@redhat.com> <5007174E.3080404@redhat.com> <500721F5.4040004@redhat.com> <500729EA.4040507@redhat.com> <500D7DE7.8080405@redhat.com> <50170A58.9030901@redhat.com> <501799C6.3090002@redhat.com> <1343831702.20530.79.camel@willson.li.ssimo.org> <1343928209.20530.132.camel@willson.li.ssimo.org> <501AB9CC.1040509@redhat.com> <501AC1A6.3090600@redhat.com> <1343937842.20530.150.camel@willson.li.ssimo.org> <1344372819.20530.314.camel@willson.li.ssimo.org> Message-ID: On Tue, Aug 7, 2012 at 1:53 PM, Simo Sorce wrote: > On Tue, 2012-08-07 at 13:30 -0700, Stephen Ingram wrote: >> On Thu, Aug 2, 2012 at 1:04 PM, Simo Sorce wrote: >> > Quick heads up in this thread, >> > >> > apparently we have isolated the issue to libkrb5 and its selinux >> > integration. >> > I have made the whole delay go away by disabling selinux entirely (which >> > makes libkrb5 stop trying to do some selinux related operations). >> > >> > We will be working on a fix to have libkrb5 not cause this delay (F17 >> > doesn't have it). >> > >> > You can follow progress on this bugzilla: >> > https://bugzilla.redhat.com/show_bug.cgi?id=845125 >> >> >> I could see this BZ for a couple of days, but now it says that I must >> log in to an account first with the appropriate permissions. Has this >> been moved or something? >> >> Steve > > Sorry Stpehen we have a stupid bot that sometimes decides to flag bugs > as internal for reasons I do not understand, I removed the flag, you > should be able to see it again now. This must have happened again as I can no longer see the bug. This must be a tough one as there seems to still be no update? Steve From rcritten at redhat.com Wed Sep 12 02:31:30 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 11 Sep 2012 22:31:30 -0400 Subject: [Freeipa-devel] slow response In-Reply-To: References: <500729EA.4040507@redhat.com> <500D7DE7.8080405@redhat.com> <50170A58.9030901@redhat.com> <501799C6.3090002@redhat.com> <1343831702.20530.79.camel@willson.li.ssimo.org> <1343928209.20530.132.camel@willson.li.ssimo.org> <501AB9CC.1040509@redhat.com> <501AC1A6.3090600@redhat.com> <1343937842.20530.150.camel@willson.li.ssimo.org> <1344372819.20530.314.camel@willson.li.ssimo.org> Message-ID: <504FF402.1040803@redhat.com> Stephen Ingram wrote: > On Tue, Aug 7, 2012 at 1:53 PM, Simo Sorce wrote: >> On Tue, 2012-08-07 at 13:30 -0700, Stephen Ingram wrote: >>> On Thu, Aug 2, 2012 at 1:04 PM, Simo Sorce wrote: >>>> Quick heads up in this thread, >>>> >>>> apparently we have isolated the issue to libkrb5 and its selinux >>>> integration. >>>> I have made the whole delay go away by disabling selinux entirely (which >>>> makes libkrb5 stop trying to do some selinux related operations). >>>> >>>> We will be working on a fix to have libkrb5 not cause this delay (F17 >>>> doesn't have it). >>>> >>>> You can follow progress on this bugzilla: >>>> https://bugzilla.redhat.com/show_bug.cgi?id=845125 >>> >>> >>> I could see this BZ for a couple of days, but now it says that I must >>> log in to an account first with the appropriate permissions. Has this >>> been moved or something? >>> >>> Steve >> >> Sorry Stpehen we have a stupid bot that sometimes decides to flag bugs >> as internal for reasons I do not understand, I removed the flag, you >> should be able to see it again now. > > This must have happened again as I can no longer see the bug. This > must be a tough one as there seems to still be no update? > Hmm, I can see it without logging into BZ. The short summary is that Nalin has a patch for it as of yesterday. rob From alee at redhat.com Wed Sep 12 02:42:26 2012 From: alee at redhat.com (Ade Lee) Date: Tue, 11 Sep 2012 22:42:26 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <504F86C0.6060907@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502B86F5.4070008@redhat.com> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> <1347329065.24731.101.camel@aleeredhat.laptop> <504F318F.3030603@redhat.com> <504F35AD.3030905@redhat.com> <1347373691.24731.125.camel@aleeredhat.laptop> <504F4CE0.9080706@redhat.com> <504F67C5.4000600@redhat.com> <504F86C0.6060907@redhat.com> Message-ID: <1347417747.24731.136.camel@aleeredhat.laptop> On Tue, 2012-09-11 at 14:45 -0400, Rob Crittenden wrote: > Petr Viktorin wrote: > > On 09/11/2012 04:38 PM, Rob Crittenden wrote: > >> Ade Lee wrote: > >>> On Tue, 2012-09-11 at 08:59 -0400, Rob Crittenden wrote: > >>>> Petr Viktorin wrote: > >>>>> On 09/11/2012 04:04 AM, Ade Lee wrote: > >>>>>> On Mon, 2012-09-10 at 16:58 -0400, Rob Crittenden wrote: > >>>>>>> Petr Viktorin wrote: > >>>>>>>> Attaching rebased and squashed patches. I've done some testing with > >>>>>>>> them > >>>>>>>> but please test some more. > >>>>>>>> > >>>>>>> > >>>>>>> Most of these aren't IPA issues, but dogtag issues. I'll try to > >>>>>>> split > >>>>>>> them out. > >>>>>>> > >>>>>>> IPA: > >>>>>>> > >>>>>>> For the configuration files in install/conf to be updated at rpm > >>>>>>> update > >>>>>>> time the VERSION needs to be incremented. > >>>>> > >>>>> These files should stay the same since on upgrade we're still using a > >>>>> Dogtag 9 style instance. The Dogtag 10 ports are only used in new > >>>>> installs. > >>>>> > >>>>>>> The ipa package lacks any updated dogtag dependencies, so I abused > >>>>>>> it. > >>>>> > >>>>> What should the updated dependencies be? Since it should work with > >>>>> both > >>>>> dogtag 9 and 10, I don't see how they should change. > >>>> > >>>> I don't know either, but we need to prevent people from installing > >>>> incompatible package combinations. > >>>> > >>> Would'nt the Conflicts: ipa < 3.0 in pki-ca mentioned below satisfy this > >>> requirement? The main concern is that you must have ipa 3.0 if you have > >>> dogtag 10. > >>> > >>> Given that dogtag is consumed by IPA though, it makes more sense to put > >>> the relevant conflicts in IPA rather than in dogtag. So in this case, > >>> that would mean putting Conflicts: pki-ca >= 10.0 in IPA 2.x. > >>> Recall that dogtag 10 will only be officially available in f18+. > >> > >> That isn't enough. If a F-17 user with IPA 2.2 installed upgrades to > >> F-18 they would be able to install dogtag 10 and blow up their IPA > >> server. > >> We can add the Conflicts: freeipa-server < 3.0 to the dogtag packages (likely in pki-base). But we should also add explicit dependencies to ipa. For ipa 2.2, Conflicts: pki-ca >= 10.0, Requires: pki-ca >= 9.x For ipa 3, Requires: pki-ca >= 10.0 This is of course assumes that ipa 3 is only officially released on f18 (which is what will happen for dogtag 10). Just because we can support d9 on ipa 3 does not mean we should. As it is, in this case, we will have to support IPA 3 + d10, IPA 3 + d10 + d9-style instance, IPA 2.x + d9. > >>> > >>>>>>> I installed IPA with dogtag 9 and created a replica. > >>>>>>> > >>>>>>> I updated the IPA bits, that worked fine. > >>>>>>> > >>>>>>> I updated to dogtag 10 and now the CA doesn't work on the master, > >>>>>>> including starting the dogtag instance. Note that the rpm update > >>>>>>> process > >>>>>>> worked, no notice that the CA service didn't restart. > >>>>>>> > >>>>>> Did you try to restart the CA with selinux in permissive mode? > >>>>>> This is > >>>>>> still required right now until I get the selinux policy all > >>>>>> straightened > >>>>>> out. > >>>>>> > >>>>>> There is also a separate dogtag ticket (which is currently being > >>>>>> worked > >>>>>> on) to restart existing dogtag instances when dogtag is upgraded from > >>>>>> 9->10. > >>>>> > >>>>> In permissive mode, this upgrade works for me. > >>>> > >>>> I was in enforcing mode but saw no AVCs. What is the ETA on fixing > >>>> this? > >>>> > >>> > >>> Within the next week or two, I need to finish the IPA merge database > >>> patch first, and then co-ordinate changes with the selinux guys. > >>> > >>>>> > >>>>> > >>>>> Sometimes I do get this error intermittently: > >>>>> > >>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to > >>>>> communicate with CMS (Service Temporarily Unavailable) > >>>>> > >>>>> Usually, waiting a couple of minutes clears this up. Perhaps we are > >>>>> not > >>>>> doing startup detection correctly. Ade mentioned that waiting for > >>>>> ports > >>>>> may not be ideal. How do we know if Dogtag is initialized? > >>>> > >>>> Years ago we had discussed with Andrew and Matt creating a URI that can > >>>> be queried to determine dogtag status. I don't think that ever went > >>>> anywhere. > >>>> > >>> Petr, this happens only on reboot, right? And not on regular "service > >>> ipa restart"? > > > > I've now seen it happen right after a 9 ? 10 upgrade. > > > >>> Yeah, I remember this conversation - and even created a bug for it at > >>> some point. This went away because the mechanism you were using seemed > >>> to be working. The timing may be very different now with tomcat 7 and > >>> systemd. I'll open a dogtag trac ticket for this. > >> > >> Ok. > >> > >>> > >>>>> > >>>>>>> Uninstalling failed because it tried to run pkidestroy and not > >>>>>>> pkiremove. > >>>>> > >>>>> I was under the impression that pkidestroy was the correct command to > >>>>> remove an upgraded instance. I'll check with Ade. > >>>>> > >>>>>> I'll test this too. > >>>>>> > >>>>>>> The contents of the file passed to pkispawn should be logged so we > >>>>>>> can > >>>>>>> see exactly what was passed in. > >>>>>>> > >>>>>> Its a pretty big file. You might want to only log the modifications. > >>>>>> Or save the file somewhere. > >>>>> > >>>>> Our logs are pretty verbose, so that shouldn't be a problem. I'll > >>>>> put it > >>>>> in the next version of the patch. > >>>> > >>>> The question to ask is: would you need the contents of this file if all > >>>> you got were logs and needed to evaluate why installation failed? In > >>>> most cases this is yes. > >>>> > >>> Up to you guys. There is a patch I am working on in which I will be > >>> logging the object that is being passed to the server from pkispawn. > >>> That - and the diffs to the standard config file as I mentioned above - > >>> will likely be sufficient to debug almost all cases. > >>> > >>> Make sure not to log any passwords. > >>> > > > > Thanks for the catch. Attaching updated patch that sanitizes the passwords. > > > >>>>>>> DOGTAG: > >>>>>>> > >>>>>>> When upgrading using the dogtag-devel repo I had to specify > >>>>>>> pki-tools.x86_64 otherwise it tried to install both 32 and 64-bit > >>>>>>> versions (and failed). > >>>>>>> > >>>>>>> I ended up running: yum update pki-ca tomcatjss pki-tools.x86_64 > >>>>>>> --enablerepo=dogtag-devel --enablerepo=updates-testing > >>>>>>> > >>>>>> We'll look into this. I think I know why this happens. > >>>>>> > >>>>>>> What happens if someone manually upgrades pki-ca without first > >>>>>>> updating > >>>>>>> ipa? I think that pki-ca is going to need a Conflicts ipa < 3.0 in > >>>>>>> it. > >>>>>> > >>>>>> We can add that. > >>>>>> > >>>>>>> certificate renewal failed. I spent far too long trying to figure > >>>>>>> out > >>>>>>> why tomcat wasn't listening on port 9180 but failed. I think 9180 is > >>>>>>> actually the old server, right? So another missing dependency on a > >>>>>>> fixed > >>>>>>> certmonger? > >>>>>>> > >>>>>>> The best I could find was the certmonger error: > >>>>>>> > >>>>>>> ca-error: Error 7 connecting to > >>>>>>> http://edsel.example.com:9180/ca/ee/ca/profileSubmit: Couldn't > >>>>>>> connect > >>>>>>> to server. > >>>>>>> > >>>>> > >>>>> I'll set the certmonger min version to 0.60 as per Nalin's mail. > >>>> > >>>> Ok. > >>>> > >>>>>> Is this cert renewal on a dogtag 10 instance? Or the upgraded > >>>>>> dogtag 9? > >>>>>> If its dogtag 10, perhaps you do not have the certmonger version that > >>>>>> has the relevant fix? If its dogtag 9, then we need to figure out > >>>>>> whats > >>>>>> going on. That reminds me - I need to file a bug to allow > >>>>>> certmonger to > >>>>>> talk to the newly defined dogtag ports. Do you have selinux > >>>>>> permissive? > >>>>>> > >>>>>>> There is no man page for pkispawn/pkidestroy :-( According to the > >>>>>>> FHS > >>>>>>> these should not be in /bin but in /usr/sbin (not end-user > >>>>>>> commands). > >>>>>>> > >>>>>> There is a trac ticket open for man pages for pkispawn and > >>>>>> pkidestroy. > >>>>>> We plan to complete this ticket by the time f18 is released. > >>>>>> > >>>>>> We'll look into the location of pkispawn/pkicreate. > >>>>>> > >>>>>>> The output of pkicreate/pkisilent was really terrible and not > >>>>>>> usable at > >>>>>>> all so we didn't display it when failures occurred. It looks like > >>>>>>> that > >>>>>>> has been addressed, at least for the case where a CA is already > >>>>>>> configured and you try to install IPA. Perhaps we should capture > >>>>>>> stderr > >>>>>>> and display that instead of the command-line of pkispawn? Again, a > >>>>>>> man > >>>>>>> page would help with the integration. > >>>>>>> > >>>>>>> 2012-09-10T20:51:45Z DEBUG [2/18]: configuring certificate server > >>>>>>> instance > >>>>>>> 2012-09-10T20:51:45Z DEBUG args=/bin/pkispawn -s CA -f > >>>>>>> /tmp/tmp_Urraq > >>>>>>> 2012-09-10T20:51:45Z DEBUG stdout= > >>>>>>> 2012-09-10T20:51:45Z DEBUG stderr=pkispawn : ERROR ....... PKI > >>>>>>> subsystem 'CA' for instance 'pki-tomcat' already exists! > >>>>>>> > >>>>>>> 2012-09-10T20:51:45Z CRITICAL failed to configure ca instance > >>>>>>> Command > >>>>>>> '/bin/pkispawn -s CA -f /tmp/tmp_Urraq' returned non-zero exit > >>>>>>> status 1 > >>>>>>> > >>>>>> That may be a good idea. Of course. thats an IPA thing, right? > >>>> > >>>> Right, not a show-stopper for this but a new ticket should be opened to > >>>> track it. > > > > https://fedorahosted.org/freeipa/ticket/3072 > > Thanks. > > I tested a 2.2 -> 3.0 upgrade with dogtag 9 and it seems to have failed > to restart something: > > [ post rpm -Uvh dist/rpms/freeipa*.rpm ] > > [root at edsel freeipa]# ipa cert-show 1 > ipa: ERROR: Certificate operation cannot be completed: Unable to > communicate with CMS (Service Temporarily Unavailable) > [root at edsel freeipa]# ipactl restart > Restarting Directory Service > Restarting KDC Service > Restarting KPASSWD Service > Restarting MEMCACHE Service > Restarting HTTP Service > Restarting CA Service > [root at edsel freeipa]# ipa cert-show 1 > Certificate: MIIDjTCCAnWgAwIBAgIBATANBgk > ... > > The apache log had this: > > [Tue Sep 11 14:35:42 2012] [error] (111)Connection refused: proxy: AJP: > attempt to connect to 127.0.0.1:9447 (localhost) failed > [Tue Sep 11 14:35:42 2012] [error] ap_proxy_connect_backend disabling > worker for (localhost) > [Tue Sep 11 14:35:42 2012] [error] proxy: AJP: failed to make connection > to backend: localhost > > So I'm gathering that dogtag didn't restart properly, but I'm just > guessing. It could have been the PKI-IPA ds instance too, I'm not sure > where to check. > > I also noticed this in /var/log/ipaupgrade.log: > > 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl start certmonger.service > 2012-09-11T18:28:22Z DEBUG stdout= > 2012-09-11T18:28:22Z DEBUG stderr= > 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl is-active certmonger.service > 2012-09-11T18:28:22Z DEBUG stdout=active > ... > ... [ snip certutil output ] > ... > 2012-09-11T18:28:52Z DEBUG args=/usr/bin/getcert start-tracking -d > /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c > dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert > "auditSigningCert cert-pki-ca" -P XXXXXXXX > 2012-09-11T18:28:52Z DEBUG stdout=Please verify that the certmonger > service is still running. > > 2012-09-11T18:28:52Z DEBUG stderr= > 2012-09-11T18:28:52Z INFO File > "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", > line 614, in run_script > return_value = main_function() > > File "/usr/sbin/ipa-upgradeconfig", line 540, in main > enable_certificate_renewal(api.env.realm) > > File "/usr/sbin/ipa-upgradeconfig", line 455, in > enable_certificate_renewal > ca.configure_renewal() > > File > "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line > 1298, in configure_renewal > self.dogtag_constants.ALIAS_DIR, 'renew_ca_cert "%s"' % nickname) > > File "/usr/lib/python2.7/site-packages/ipapython/certmonger.py", line > 394, in dogtag_start_tracking > (stdout, stderr, returncode) = ipautil.run(args, nolog=[pin]) > > File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line > 309, in run > raise CalledProcessError(p.returncode, args) > > 2012-09-11T18:28:52Z INFO The ipa-upgradeconfig command failed, > exception: CalledProcessError: Command '/usr/bin/getcert start-tracking > -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c > dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert > "auditSigningCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit status 1 > > I'm not sure if this is related to this patch or not. If it isn't can > you file a new ticket on it, or add it the 2.2 -> 3.0 upgrade ticket? > > rob > From sbingram at gmail.com Wed Sep 12 04:17:01 2012 From: sbingram at gmail.com (Stephen Ingram) Date: Tue, 11 Sep 2012 21:17:01 -0700 Subject: [Freeipa-devel] slow response In-Reply-To: <504FF402.1040803@redhat.com> References: <500729EA.4040507@redhat.com> <500D7DE7.8080405@redhat.com> <50170A58.9030901@redhat.com> <501799C6.3090002@redhat.com> <1343831702.20530.79.camel@willson.li.ssimo.org> <1343928209.20530.132.camel@willson.li.ssimo.org> <501AB9CC.1040509@redhat.com> <501AC1A6.3090600@redhat.com> <1343937842.20530.150.camel@willson.li.ssimo.org> <1344372819.20530.314.camel@willson.li.ssimo.org> <504FF402.1040803@redhat.com> Message-ID: On Tue, Sep 11, 2012 at 7:31 PM, Rob Crittenden wrote: > Stephen Ingram wrote: >> >> On Tue, Aug 7, 2012 at 1:53 PM, Simo Sorce wrote: >>> >>> On Tue, 2012-08-07 at 13:30 -0700, Stephen Ingram wrote: >>>> >>>> On Thu, Aug 2, 2012 at 1:04 PM, Simo Sorce wrote: >>>>> >>>>> Quick heads up in this thread, >>>>> >>>>> apparently we have isolated the issue to libkrb5 and its selinux >>>>> integration. >>>>> I have made the whole delay go away by disabling selinux entirely >>>>> (which >>>>> makes libkrb5 stop trying to do some selinux related operations). >>>>> >>>>> We will be working on a fix to have libkrb5 not cause this delay (F17 >>>>> doesn't have it). >>>>> >>>>> You can follow progress on this bugzilla: >>>>> https://bugzilla.redhat.com/show_bug.cgi?id=845125 >>>> >>>> >>>> >>>> I could see this BZ for a couple of days, but now it says that I must >>>> log in to an account first with the appropriate permissions. Has this >>>> been moved or something? >>>> >>>> Steve >>> >>> >>> Sorry Stpehen we have a stupid bot that sometimes decides to flag bugs >>> as internal for reasons I do not understand, I removed the flag, you >>> should be able to see it again now. >> >> >> This must have happened again as I can no longer see the bug. This >> must be a tough one as there seems to still be no update? >> > > Hmm, I can see it without logging into BZ. The short summary is that Nalin > has a patch for it as of yesterday. I can see the bug again now too. Strange. Thanks for the update too. Steve From pvoborni at redhat.com Wed Sep 12 07:42:32 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Wed, 12 Sep 2012 09:42:32 +0200 Subject: [Freeipa-devel] [PATCH] 211 Prevent opening of multiple dirty dialogs on navigation In-Reply-To: <504E3B86.1040204@redhat.com> References: <5049EA77.9010308@redhat.com> <504E3B86.1040204@redhat.com> Message-ID: <50503CE8.5050102@redhat.com> On 09/10/2012 09:12 PM, Endi Sukma Dewata wrote: > On 9/7/2012 7:37 AM, Petr Vobornik wrote: >> Facets which performs AJAX call after update refresh (clear dirty state) >> after calling callback of dirty dialog. It might lead to multiple >> openings of dirty dialog. >> >> Assuming that calling dirty dialog's callback can be evaluated as "dirty >> state is gone", we can call reset in the callback to prevent the issue. >> There will be an incorrect state in the facet for a moment. It will be >> fixed soon on execute of callback of the refresh AJAX call. It is not an >> issue because it will happen in background. User will be looking on >> different facet. >> >> https://fedorahosted.org/freeipa/ticket/2667 > > ACK. > Pushed to master, ipa-3-0. -- Petr Vobornik From tbabej at redhat.com Wed Sep 12 08:24:56 2012 From: tbabej at redhat.com (Tomas Babej) Date: Wed, 12 Sep 2012 10:24:56 +0200 Subject: [Freeipa-devel] [PATCH 0011] Make sure selinuxusemap behaves consistently to HBAC rule In-Reply-To: <504F1D1F.8070505@redhat.com> References: <5044C8CE.20108@redhat.com> <50473DDF.8040303@redhat.com> <50488561.7090902@redhat.com> <504F1D1F.8070505@redhat.com> Message-ID: <505046D8.8030204@redhat.com> On 09/11/2012 01:14 PM, Martin Kosek wrote: > On 09/06/2012 01:13 PM, Tomas Babej wrote: >> On 09/05/2012 01:56 PM, Martin Kosek wrote: >>> On 09/03/2012 05:12 PM, Tomas Babej wrote: >>>> Hi, >>>> >>>> Both selinuxusermap-add and selinuxusermap-mod commands now behave >>>> consistently in not allowing user/host category or user/host members >>>> and HBAC rule being set at the same time. Also adds a bunch of unit >>>> tests that check this behaviour. >>>> >>>> https://fedorahosted.org/freeipa/ticket/2983 >>>> >>>> Tomas >>>> >>> I found few issues with this patch: >>> >>> 1) Patch needs a rebase >>> >>> 2) Patch does not expect attributes to be set to None, i.e. to be left empty or >>> to be deleted, e.g.: >>> >>> # ipa selinuxusermap-add foo --selinuxuser=guest_u:s0 --usercat=all --hbacrule= >>> ipa: ERROR: HBAC rule and local members cannot both be set >>> >>> # ipa selinuxusermap-add foo --selinuxuser=guest_u:s0 --usercat=all >>> ---------------------------- >>> Added SELinux User Map "foo" >>> ---------------------------- >>> Rule name: foo >>> SELinux User: guest_u:s0 >>> User category: all >>> Enabled: TRUE >>> >>> # ipa selinuxusermap-mod foo --usercat= --hbacrule= >>> ipa: ERROR: HBAC rule and local members cannot both be set >>> >>> # ipa selinuxusermap-mod foo --usercat= >>> ------------------------------- >>> Modified SELinux User Map "foo" >>> ------------------------------- >>> Rule name: foo >>> SELinux User: guest_u:s0 >>> Enabled: TRUE >>> >>> # ipa selinuxusermap-mod foo --hbacrule=foo >>> ------------------------------- >>> Modified SELinux User Map "foo" >>> ------------------------------- >>> Rule name: foo >>> SELinux User: guest_u:s0 >>> HBAC Rule: foo >>> Enabled: TRUE >>> >>> # ipa selinuxusermap-mod foo --hbacrule= --usercat=all >>> ipa: ERROR: HBAC rule and local members cannot both be set >>> >>> All these validation failures are not valid. >>> >>> 3) Additionally, I think it would be more readable and less error prone that if >>> instead of this blob: >>> >>> + are_local_members_to_be_set = 'usercategory' in _entry_attrs or \ >>> + 'hostcategory' in _entry_attrs or \ >>> + 'memberuser' in _entry_attrs or \ >>> + 'memberhost' in _entry_attrs >>> >>> You would use something like that: >>> >>> are_local_members_to_be_set = any(attr in _entry_attrs >>> for attr in ('usercategory', >>> 'hostcategory', >>> 'memberuser', >>> 'memberhost')) >>> >>> Martin >> 1.) Done. >> 2.) Corrected. >> 3.) Fixed. >> >> Tomas > 1) There are some (corner) cases where this approach still does not work: > > # ipa selinuxusermap-show foo > Rule name: foo > SELinux User: guest_u:s0 > HBAC Rule: foo > Enabled: TRUE > # ipa selinuxusermap-mod foo --usercat=all --hbacrule= > ipa: ERROR: HBAC rule and local members cannot both be set > > HBAC rule attribute is being deleted and user category set, so this should not > be rejected. > > 2) There are also some styling issues (you can use pep8 tool present in Fedora > to locate them on your own, e.g.: > > ipalib/plugins/selinuxusermap.py:247:32: E203 whitespace before ':' > ipalib/plugins/selinuxusermap.py:247:70: E225 missing whitespace around operator > ipalib/plugins/selinuxusermap.py:249:36: E221 multiple spaces before operator > ... > > Martin The corner case is fixed now and styling issues corrected as well. Tomas -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-tbabej-0011-3-Make-sure-selinuxusemap-behaves-consistently-to-HBAC.patch Type: text/x-patch Size: 15148 bytes Desc: not available URL: From pviktori at redhat.com Wed Sep 12 08:42:23 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Wed, 12 Sep 2012 10:42:23 +0200 Subject: [Freeipa-devel] [PATCH] 0079 Update the pot file (translation source) Message-ID: <50504AEF.9090209@redhat.com> Transifex is watching our repository, so pushing this patch will update the translations on the site. -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0079-Update-the-pot-file-translation-source.patch Type: text/x-patch Size: 194785 bytes Desc: not available URL: From pspacek at redhat.com Wed Sep 12 10:33:47 2012 From: pspacek at redhat.com (Petr Spacek) Date: Wed, 12 Sep 2012 12:33:47 +0200 Subject: [Freeipa-devel] [PATCH 0056] Fix crash caused by zone deletion vs. SOA serial increment race condition Message-ID: <5050650B.3000709@redhat.com> Hello, The patch fixes crash caused by stupid bug in logging code. Petr^2 Spacek -------------- next part -------------- A non-text attachment was scrubbed... Name: bind-dyndb-ldap-pspacek-0056-Fix-crash-caused-by-zone-deletion-vs.-SOA-serial-inc.patch Type: text/x-patch Size: 1545 bytes Desc: not available URL: From pspacek at redhat.com Wed Sep 12 10:35:25 2012 From: pspacek at redhat.com (Petr Spacek) Date: Wed, 12 Sep 2012 12:35:25 +0200 Subject: [Freeipa-devel] [PATCH 0057] Fix LDAP operation selection logic in ldap_modify_do() Message-ID: <5050656D.8010604@redhat.com> Hello, There is a fix for LDAP operation selection logic in ldap_modify_do(). Each operation code in LDAPMod structure can be ORed with LDAP_MOD_BVALUES. Petr^2 Spacek -------------- next part -------------- A non-text attachment was scrubbed... Name: bind-dyndb-ldap-pspacek-0057-Fix-LDAP-operation-selection-logic-in-ldap_modify_do.patch Type: text/x-patch Size: 2900 bytes Desc: not available URL: From pspacek at redhat.com Wed Sep 12 10:36:38 2012 From: pspacek at redhat.com (Petr Spacek) Date: Wed, 12 Sep 2012 12:36:38 +0200 Subject: [Freeipa-devel] [PATCH 0058] Improve persistent search logging Message-ID: <505065B6.50004@redhat.com> Hello, this patch adds "result" codes to error messages in persistent search code. Petr^2 Spacek -------------- next part -------------- A non-text attachment was scrubbed... Name: bind-dyndb-ldap-pspacek-0058-Improve-persistent-search-logging.patch Type: text/x-patch Size: 2290 bytes Desc: not available URL: From pspacek at redhat.com Wed Sep 12 11:07:56 2012 From: pspacek at redhat.com (Petr Spacek) Date: Wed, 12 Sep 2012 13:07:56 +0200 Subject: [Freeipa-devel] [PATCH 0059] Fix potential crash after free(uninitialized variable) Message-ID: <50506D0C.70706@redhat.com> Hello, This patch fixes potential crash after free(uninitialized variable) in persistent search code. Coverity CID 13088. Petr^2 Spacek -------------- next part -------------- A non-text attachment was scrubbed... Name: bind-dyndb-ldap-pspacek-0059-Fix-potential-crash-after-free-uninitialized-variabl.patch Type: text/x-patch Size: 1269 bytes Desc: not available URL: From pviktori at redhat.com Wed Sep 12 11:20:03 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Wed, 12 Sep 2012 13:20:03 +0200 Subject: [Freeipa-devel] [PATCH] 0078 ipa-client-install: Obtain host TGT from one specific KDC In-Reply-To: <504FA192.5070000@redhat.com> References: <5048929E.7080605@redhat.com> <504FA192.5070000@redhat.com> Message-ID: <50506FE3.9070505@redhat.com> On 09/11/2012 10:39 PM, Rob Crittenden wrote: > Petr Viktorin wrote: >> When installing the client, we need to take extra case to only contact >> the one server we're installing against. Otherwise, in the real world, >> we might hit a server that hasn't replicated info about the client yet. >> >> This patch fixes a bug where kinit attempted to contact a KDC that >> didn't have the host principal yet. >> >> >> To reproduce: >> >> - Install a "master" and "replica" >> - Change the Kerberos DNS entries to only point to the replica: >> for REC_NAME in '_kerberos-master._tcp' '_kerberos-master._udp' >> '_kerberos._tcp' '_kerberos._udp' '_kpasswd._tcp' '_kpasswd._udp'; do >> ipa dnsrecord-mod $DOMAIN $REC_NAME --srv-rec="0 100 88 >> $REPLICA_HOSTNAME" >> done >> ipa dnsrecord-mod $DOMAIN _ldap._tcp --srv-rec="0 100 389 >> $MASTER_HOSTNAME" >> ipa dnsrecord-find $DOMAIN # check >> - Sever communication between the hosts to disable replication: >> (on master) >> iptables -A INPUT -j DROP -p all --source $REPLICA_IP >> - On client machine, put master as nameserver in /etc/resolv.conf & >> install client >> >> This will fail without the patch. >> >> >> Thanks to Petr Spacek, Simo, and Scott for helping to reproduce and >> explain the bug. I learned a lot. >> >> https://fedorahosted.org/freeipa/ticket/2982 > > ACK, pushed to master and ipa-3-0 > > rob > The patch broke server installs. Please revert it if you're having trouble while I look into it. -- Petr? From pviktori at redhat.com Wed Sep 12 12:09:15 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Wed, 12 Sep 2012 14:09:15 +0200 Subject: [Freeipa-devel] [PATCH] 0078 ipa-client-install: Obtain host TGT from one specific KDC In-Reply-To: <50506FE3.9070505@redhat.com> References: <5048929E.7080605@redhat.com> <504FA192.5070000@redhat.com> <50506FE3.9070505@redhat.com> Message-ID: <50507B6B.7050908@redhat.com> On 09/12/2012 01:20 PM, Petr Viktorin wrote: > On 09/11/2012 10:39 PM, Rob Crittenden wrote: >> Petr Viktorin wrote: >>> When installing the client, we need to take extra case to only contact >>> the one server we're installing against. Otherwise, in the real world, >>> we might hit a server that hasn't replicated info about the client yet. >>> >>> This patch fixes a bug where kinit attempted to contact a KDC that >>> didn't have the host principal yet. >>> >>> >>> To reproduce: >>> >>> - Install a "master" and "replica" >>> - Change the Kerberos DNS entries to only point to the replica: >>> for REC_NAME in '_kerberos-master._tcp' '_kerberos-master._udp' >>> '_kerberos._tcp' '_kerberos._udp' '_kpasswd._tcp' '_kpasswd._udp'; do >>> ipa dnsrecord-mod $DOMAIN $REC_NAME --srv-rec="0 100 88 >>> $REPLICA_HOSTNAME" >>> done >>> ipa dnsrecord-mod $DOMAIN _ldap._tcp --srv-rec="0 100 389 >>> $MASTER_HOSTNAME" >>> ipa dnsrecord-find $DOMAIN # check >>> - Sever communication between the hosts to disable replication: >>> (on master) >>> iptables -A INPUT -j DROP -p all --source $REPLICA_IP >>> - On client machine, put master as nameserver in /etc/resolv.conf & >>> install client >>> >>> This will fail without the patch. >>> >>> >>> Thanks to Petr Spacek, Simo, and Scott for helping to reproduce and >>> explain the bug. I learned a lot. >>> >>> https://fedorahosted.org/freeipa/ticket/2982 >> >> ACK, pushed to master and ipa-3-0 >> >> rob >> > > The patch broke server installs. Please revert it if you're having > trouble while I look into it. > > I messed up and removed the kinit call entirely when installing on master. Attaching a fix. -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0080-Fix-server-installation.patch Type: text/x-patch Size: 1570 bytes Desc: not available URL: From mkosek at redhat.com Wed Sep 12 12:08:17 2012 From: mkosek at redhat.com (Martin Kosek) Date: Wed, 12 Sep 2012 14:08:17 +0200 Subject: [Freeipa-devel] [PATCH] 305-308 Expand Referential Integrity checks Message-ID: <50507B31.6070804@redhat.com> To test, add sudo commands, hosts or users to a sudo rule or hbac rule and then rename or delete the linked object. After the update, the links should be amended. --------- Many attributes in IPA (e.g. manager, memberuser, managedby, ...) are used to store DNs of linked objects in IPA (users, hosts, sudo commands, etc.). However, when the linked objects is deleted or renamed, the attribute pointing to it stays with the objects and thus may create a dangling link causing issues in client software reading the data. Directory Server has a plugin to enforce referential integrity (RI) by checking DEL and MODRDN operations and updating affected links. It was already used for manager and secretary attributes and should be expanded for the missing attributes to avoid dangling links. As a prerequisite, all attributes checked for RI must have pres and eq indexes to avoid performance issues. The following indexes have been added: * manager (pres index only) * secretary (pres index only) * memberHost * memberUser * sourcehost * memberservice * managedby * memberallowcmd * memberdenycmd * ipasudorunas * ipasudorunasgroup Referential Integrity plugin was updated to check all these attributes. Note: this update will only fix RI on one master as RI plugin does not check replicated operations. https://fedorahosted.org/freeipa/ticket/2866 -- Martin Kosek Senior Software Engineer - Identity Management Team Red Hat Inc. -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-305-add-attributeTypes-to-safe-schema-updater.patch Type: text/x-patch Size: 6467 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-306-amend-memberallowcmd-and-memberdenycmd-attribute-typ.patch Type: text/x-patch Size: 4864 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-307-run-index-task-in-ldap-updater-only-when-needed.patch Type: text/x-patch Size: 2813 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-308-expand-referential-integrity-checks.patch Type: text/x-patch Size: 11193 bytes Desc: not available URL: From pviktori at redhat.com Wed Sep 12 12:46:48 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Wed, 12 Sep 2012 14:46:48 +0200 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <1347417747.24731.136.camel@aleeredhat.laptop> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> <1347329065.24731.101.camel@aleeredhat.laptop> <504F318F.3030603@redhat.com> <504F35AD.3030905@redhat.com> <1347373691.24731.125.camel@aleeredhat.laptop> <504F4CE0.9080706@redhat.com> <504F67C5.4000600@redhat.com> <504F86C0.6060907@redhat.com> <1347417747.24731.136.camel@aleeredhat.laptop> Message-ID: <50508438.7090101@redhat.com> On 09/12/2012 04:42 AM, Ade Lee wrote: > On Tue, 2012-09-11 at 14:45 -0400, Rob Crittenden wrote: >> Petr Viktorin wrote: >>> On 09/11/2012 04:38 PM, Rob Crittenden wrote: >>>> Ade Lee wrote: >>>>> On Tue, 2012-09-11 at 08:59 -0400, Rob Crittenden wrote: >>>>>> Petr Viktorin wrote: >>>>>>> On 09/11/2012 04:04 AM, Ade Lee wrote: >>>>>>>> On Mon, 2012-09-10 at 16:58 -0400, Rob Crittenden wrote: >>>>>>>>> Petr Viktorin wrote: >>>>>>>>>> Attaching rebased and squashed patches. I've done some testing with >>>>>>>>>> them >>>>>>>>>> but please test some more. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Most of these aren't IPA issues, but dogtag issues. I'll try to >>>>>>>>> split >>>>>>>>> them out. >>>>>>>>> >>>>>>>>> IPA: >>>>>>>>> >>>>>>>>> For the configuration files in install/conf to be updated at rpm >>>>>>>>> update >>>>>>>>> time the VERSION needs to be incremented. >>>>>>> >>>>>>> These files should stay the same since on upgrade we're still using a >>>>>>> Dogtag 9 style instance. The Dogtag 10 ports are only used in new >>>>>>> installs. >>>>>>> >>>>>>>>> The ipa package lacks any updated dogtag dependencies, so I abused >>>>>>>>> it. >>>>>>> >>>>>>> What should the updated dependencies be? Since it should work with >>>>>>> both >>>>>>> dogtag 9 and 10, I don't see how they should change. >>>>>> >>>>>> I don't know either, but we need to prevent people from installing >>>>>> incompatible package combinations. >>>>>> >>>>> Would'nt the Conflicts: ipa < 3.0 in pki-ca mentioned below satisfy this >>>>> requirement? The main concern is that you must have ipa 3.0 if you have >>>>> dogtag 10. >>>>> >>>>> Given that dogtag is consumed by IPA though, it makes more sense to put >>>>> the relevant conflicts in IPA rather than in dogtag. So in this case, >>>>> that would mean putting Conflicts: pki-ca >= 10.0 in IPA 2.x. >>>>> Recall that dogtag 10 will only be officially available in f18+. >>>> >>>> That isn't enough. If a F-17 user with IPA 2.2 installed upgrades to >>>> F-18 they would be able to install dogtag 10 and blow up their IPA >>>> server. >>>> > We can add the Conflicts: freeipa-server < 3.0 to the dogtag packages > (likely in pki-base). > > But we should also add explicit dependencies to ipa. > > For ipa 2.2, Conflicts: pki-ca >= 10.0, Requires: pki-ca >= 9.x > For ipa 3, Requires: pki-ca >= 10.0 Unfortunately we need to support IPA 3.0 with Dogtag 9. > This is of course assumes that ipa 3 is only officially released on f18 > (which is what will happen for dogtag 10). Just because we can support > d9 on ipa 3 does not mean we should. > > As it is, in this case, we will have to support IPA 3 + d10, IPA 3 + d10 > + d9-style instance, IPA 2.x + d9. We also need to test replication between various combinations of these. -- Petr? From jcholast at redhat.com Wed Sep 12 12:58:18 2012 From: jcholast at redhat.com (Jan Cholasta) Date: Wed, 12 Sep 2012 14:58:18 +0200 Subject: [Freeipa-devel] [PATCH] 0078 ipa-client-install: Obtain host TGT from one specific KDC In-Reply-To: <50507B6B.7050908@redhat.com> References: <5048929E.7080605@redhat.com> <504FA192.5070000@redhat.com> <50506FE3.9070505@redhat.com> <50507B6B.7050908@redhat.com> Message-ID: <505086EA.50402@redhat.com> Dne 12.9.2012 14:09, Petr Viktorin napsal(a): > On 09/12/2012 01:20 PM, Petr Viktorin wrote: >> On 09/11/2012 10:39 PM, Rob Crittenden wrote: >>> Petr Viktorin wrote: >>>> When installing the client, we need to take extra case to only contact >>>> the one server we're installing against. Otherwise, in the real world, >>>> we might hit a server that hasn't replicated info about the client yet. >>>> >>>> This patch fixes a bug where kinit attempted to contact a KDC that >>>> didn't have the host principal yet. >>>> >>>> >>>> To reproduce: >>>> >>>> - Install a "master" and "replica" >>>> - Change the Kerberos DNS entries to only point to the replica: >>>> for REC_NAME in '_kerberos-master._tcp' '_kerberos-master._udp' >>>> '_kerberos._tcp' '_kerberos._udp' '_kpasswd._tcp' '_kpasswd._udp'; do >>>> ipa dnsrecord-mod $DOMAIN $REC_NAME --srv-rec="0 100 88 >>>> $REPLICA_HOSTNAME" >>>> done >>>> ipa dnsrecord-mod $DOMAIN _ldap._tcp --srv-rec="0 100 389 >>>> $MASTER_HOSTNAME" >>>> ipa dnsrecord-find $DOMAIN # check >>>> - Sever communication between the hosts to disable replication: >>>> (on master) >>>> iptables -A INPUT -j DROP -p all --source $REPLICA_IP >>>> - On client machine, put master as nameserver in /etc/resolv.conf & >>>> install client >>>> >>>> This will fail without the patch. >>>> >>>> >>>> Thanks to Petr Spacek, Simo, and Scott for helping to reproduce and >>>> explain the bug. I learned a lot. >>>> >>>> https://fedorahosted.org/freeipa/ticket/2982 >>> >>> ACK, pushed to master and ipa-3-0 >>> >>> rob >>> >> >> The patch broke server installs. Please revert it if you're having >> trouble while I look into it. >> >> > > I messed up and removed the kinit call entirely when installing on > master. Attaching a fix. > Works for me, ACK. Honza -- Jan Cholasta From rcritten at redhat.com Wed Sep 12 12:58:37 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 12 Sep 2012 08:58:37 -0400 Subject: [Freeipa-devel] [PATCH] 0078 ipa-client-install: Obtain host TGT from one specific KDC In-Reply-To: <505086EA.50402@redhat.com> References: <5048929E.7080605@redhat.com> <504FA192.5070000@redhat.com> <50506FE3.9070505@redhat.com> <50507B6B.7050908@redhat.com> <505086EA.50402@redhat.com> Message-ID: <505086FD.9060403@redhat.com> Jan Cholasta wrote: > Dne 12.9.2012 14:09, Petr Viktorin napsal(a): >> On 09/12/2012 01:20 PM, Petr Viktorin wrote: >>> On 09/11/2012 10:39 PM, Rob Crittenden wrote: >>>> Petr Viktorin wrote: >>>>> When installing the client, we need to take extra case to only contact >>>>> the one server we're installing against. Otherwise, in the real world, >>>>> we might hit a server that hasn't replicated info about the client >>>>> yet. >>>>> >>>>> This patch fixes a bug where kinit attempted to contact a KDC that >>>>> didn't have the host principal yet. >>>>> >>>>> >>>>> To reproduce: >>>>> >>>>> - Install a "master" and "replica" >>>>> - Change the Kerberos DNS entries to only point to the replica: >>>>> for REC_NAME in '_kerberos-master._tcp' '_kerberos-master._udp' >>>>> '_kerberos._tcp' '_kerberos._udp' '_kpasswd._tcp' '_kpasswd._udp'; do >>>>> ipa dnsrecord-mod $DOMAIN $REC_NAME --srv-rec="0 100 88 >>>>> $REPLICA_HOSTNAME" >>>>> done >>>>> ipa dnsrecord-mod $DOMAIN _ldap._tcp --srv-rec="0 100 389 >>>>> $MASTER_HOSTNAME" >>>>> ipa dnsrecord-find $DOMAIN # check >>>>> - Sever communication between the hosts to disable replication: >>>>> (on master) >>>>> iptables -A INPUT -j DROP -p all --source $REPLICA_IP >>>>> - On client machine, put master as nameserver in /etc/resolv.conf & >>>>> install client >>>>> >>>>> This will fail without the patch. >>>>> >>>>> >>>>> Thanks to Petr Spacek, Simo, and Scott for helping to reproduce and >>>>> explain the bug. I learned a lot. >>>>> >>>>> https://fedorahosted.org/freeipa/ticket/2982 >>>> >>>> ACK, pushed to master and ipa-3-0 >>>> >>>> rob >>>> >>> >>> The patch broke server installs. Please revert it if you're having >>> trouble while I look into it. >>> >>> >> >> I messed up and removed the kinit call entirely when installing on >> master. Attaching a fix. >> > > Works for me, ACK. > > Honza > pushed to master and ipa-3-0 From jcholast at redhat.com Wed Sep 12 13:09:36 2012 From: jcholast at redhat.com (Jan Cholasta) Date: Wed, 12 Sep 2012 15:09:36 +0200 Subject: [Freeipa-devel] [PATCH] 84 Add the SSH service to SSSD config file before trying to activate it Message-ID: <50508990.8070700@redhat.com> Hi, this patch fixes . Users no longer have to configure SSH in sssd.conf manually if the file exists prior to running ipa-client-install. Honza -- Jan Cholasta -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-jcholast-84-add-ssh-service-to-sssd-conf.patch Type: text/x-patch Size: 1443 bytes Desc: not available URL: From jcholast at redhat.com Wed Sep 12 13:53:45 2012 From: jcholast at redhat.com (Jan Cholasta) Date: Wed, 12 Sep 2012 15:53:45 +0200 Subject: [Freeipa-devel] [PATCH] 85 Add --no-ssh option to ipa-client-install to disable OpenSSH client configuration Message-ID: <505093E9.3070608@redhat.com> Hi, this patch fixes . If both --no-ssh and --no-sshd are specified, do not configure the SSH service in SSSD. Honza -- Jan Cholasta -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-jcholast-85-Add-no-ssh-option-to-ipa-client-install-to-disable-O.patch Type: text/x-patch Size: 9854 bytes Desc: not available URL: From mkosek at redhat.com Wed Sep 12 14:04:37 2012 From: mkosek at redhat.com (Martin Kosek) Date: Wed, 12 Sep 2012 16:04:37 +0200 Subject: [Freeipa-devel] [PATCH] 0078 ipa-client-install: Obtain host TGT from one specific KDC In-Reply-To: <505086EA.50402@redhat.com> References: <5048929E.7080605@redhat.com> <504FA192.5070000@redhat.com> <50506FE3.9070505@redhat.com> <50507B6B.7050908@redhat.com> <505086EA.50402@redhat.com> Message-ID: <50509675.4010304@redhat.com> On 09/12/2012 02:58 PM, Jan Cholasta wrote: > Dne 12.9.2012 14:09, Petr Viktorin napsal(a): >> On 09/12/2012 01:20 PM, Petr Viktorin wrote: >>> On 09/11/2012 10:39 PM, Rob Crittenden wrote: >>>> Petr Viktorin wrote: >>>>> When installing the client, we need to take extra case to only contact >>>>> the one server we're installing against. Otherwise, in the real world, >>>>> we might hit a server that hasn't replicated info about the client yet. >>>>> >>>>> This patch fixes a bug where kinit attempted to contact a KDC that >>>>> didn't have the host principal yet. >>>>> >>>>> >>>>> To reproduce: >>>>> >>>>> - Install a "master" and "replica" >>>>> - Change the Kerberos DNS entries to only point to the replica: >>>>> for REC_NAME in '_kerberos-master._tcp' '_kerberos-master._udp' >>>>> '_kerberos._tcp' '_kerberos._udp' '_kpasswd._tcp' '_kpasswd._udp'; do >>>>> ipa dnsrecord-mod $DOMAIN $REC_NAME --srv-rec="0 100 88 >>>>> $REPLICA_HOSTNAME" >>>>> done >>>>> ipa dnsrecord-mod $DOMAIN _ldap._tcp --srv-rec="0 100 389 >>>>> $MASTER_HOSTNAME" >>>>> ipa dnsrecord-find $DOMAIN # check >>>>> - Sever communication between the hosts to disable replication: >>>>> (on master) >>>>> iptables -A INPUT -j DROP -p all --source $REPLICA_IP >>>>> - On client machine, put master as nameserver in /etc/resolv.conf & >>>>> install client >>>>> >>>>> This will fail without the patch. >>>>> >>>>> >>>>> Thanks to Petr Spacek, Simo, and Scott for helping to reproduce and >>>>> explain the bug. I learned a lot. >>>>> >>>>> https://fedorahosted.org/freeipa/ticket/2982 >>>> >>>> ACK, pushed to master and ipa-3-0 >>>> >>>> rob >>>> >>> >>> The patch broke server installs. Please revert it if you're having >>> trouble while I look into it. >>> >>> >> >> I messed up and removed the kinit call entirely when installing on >> master. Attaching a fix. >> > > Works for me, ACK. > > Honza > When the server installation is complete, I was surprised to see I have now host credentials in my CCACHE: # ipa-server-install --setup-dns ... ============================================================================== Setup complete Next steps: 1. You must make sure these network ports are open: TCP Ports: * 80, 443: HTTP/HTTPS * 389, 636: LDAP/LDAPS * 88, 464: kerberos * 53: bind UDP Ports: * 88, 464: kerberos * 53: bind * 123: ntp 2. You can now obtain a kerberos ticket using the command: 'kinit admin' This ticket will allow you to use the IPA tools (e.g., ipa user-add) and the web user interface. Be sure to back up the CA certificate stored in /root/cacert.p12 This file is required to create replicas. The password for this file is the Directory Manager password # klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: host/vm-086.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM Valid starting Expires Service principal 09/12/12 09:28:24 09/13/12 09:28:24 krbtgt/IDM.LAB.BOS.REDHAT.COM at IDM.LAB.BOS.REDHAT.COM 09/12/12 09:28:24 09/13/12 09:28:24 HTTP/vm-086.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM 09/12/12 09:28:26 09/13/12 09:28:24 DNS/vm-086.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM I don't think this is an expected behavior, installer should use a CCACHE separate from user's default. Martin From mkosek at redhat.com Wed Sep 12 14:14:09 2012 From: mkosek at redhat.com (Martin Kosek) Date: Wed, 12 Sep 2012 16:14:09 +0200 Subject: [Freeipa-devel] [PATCH 0011] Make sure selinuxusemap behaves consistently to HBAC rule In-Reply-To: <505046D8.8030204@redhat.com> References: <5044C8CE.20108@redhat.com> <50473DDF.8040303@redhat.com> <50488561.7090902@redhat.com> <504F1D1F.8070505@redhat.com> <505046D8.8030204@redhat.com> Message-ID: <505098B1.8040903@redhat.com> On 09/12/2012 10:24 AM, Tomas Babej wrote: > On 09/11/2012 01:14 PM, Martin Kosek wrote: >> On 09/06/2012 01:13 PM, Tomas Babej wrote: >>> On 09/05/2012 01:56 PM, Martin Kosek wrote: >>>> On 09/03/2012 05:12 PM, Tomas Babej wrote: >>>>> Hi, >>>>> >>>>> Both selinuxusermap-add and selinuxusermap-mod commands now behave >>>>> consistently in not allowing user/host category or user/host members >>>>> and HBAC rule being set at the same time. Also adds a bunch of unit >>>>> tests that check this behaviour. >>>>> >>>>> https://fedorahosted.org/freeipa/ticket/2983 >>>>> >>>>> Tomas >>>>> >>>> I found few issues with this patch: >>>> >>>> 1) Patch needs a rebase >>>> >>>> 2) Patch does not expect attributes to be set to None, i.e. to be left >>>> empty or >>>> to be deleted, e.g.: >>>> >>>> # ipa selinuxusermap-add foo --selinuxuser=guest_u:s0 --usercat=all >>>> --hbacrule= >>>> ipa: ERROR: HBAC rule and local members cannot both be set >>>> >>>> # ipa selinuxusermap-add foo --selinuxuser=guest_u:s0 --usercat=all >>>> ---------------------------- >>>> Added SELinux User Map "foo" >>>> ---------------------------- >>>> Rule name: foo >>>> SELinux User: guest_u:s0 >>>> User category: all >>>> Enabled: TRUE >>>> >>>> # ipa selinuxusermap-mod foo --usercat= --hbacrule= >>>> ipa: ERROR: HBAC rule and local members cannot both be set >>>> >>>> # ipa selinuxusermap-mod foo --usercat= >>>> ------------------------------- >>>> Modified SELinux User Map "foo" >>>> ------------------------------- >>>> Rule name: foo >>>> SELinux User: guest_u:s0 >>>> Enabled: TRUE >>>> >>>> # ipa selinuxusermap-mod foo --hbacrule=foo >>>> ------------------------------- >>>> Modified SELinux User Map "foo" >>>> ------------------------------- >>>> Rule name: foo >>>> SELinux User: guest_u:s0 >>>> HBAC Rule: foo >>>> Enabled: TRUE >>>> >>>> # ipa selinuxusermap-mod foo --hbacrule= --usercat=all >>>> ipa: ERROR: HBAC rule and local members cannot both be set >>>> >>>> All these validation failures are not valid. >>>> >>>> 3) Additionally, I think it would be more readable and less error prone >>>> that if >>>> instead of this blob: >>>> >>>> + are_local_members_to_be_set = 'usercategory' in _entry_attrs or \ >>>> + 'hostcategory' in _entry_attrs or \ >>>> + 'memberuser' in _entry_attrs or \ >>>> + 'memberhost' in _entry_attrs >>>> >>>> You would use something like that: >>>> >>>> are_local_members_to_be_set = any(attr in _entry_attrs >>>> for attr in ('usercategory', >>>> 'hostcategory', >>>> 'memberuser', >>>> 'memberhost')) >>>> >>>> Martin >>> 1.) Done. >>> 2.) Corrected. >>> 3.) Fixed. >>> >>> Tomas >> 1) There are some (corner) cases where this approach still does not work: >> >> # ipa selinuxusermap-show foo >> Rule name: foo >> SELinux User: guest_u:s0 >> HBAC Rule: foo >> Enabled: TRUE >> # ipa selinuxusermap-mod foo --usercat=all --hbacrule= >> ipa: ERROR: HBAC rule and local members cannot both be set >> >> HBAC rule attribute is being deleted and user category set, so this should not >> be rejected. >> >> 2) There are also some styling issues (you can use pep8 tool present in Fedora >> to locate them on your own, e.g.: >> >> ipalib/plugins/selinuxusermap.py:247:32: E203 whitespace before ':' >> ipalib/plugins/selinuxusermap.py:247:70: E225 missing whitespace around operator >> ipalib/plugins/selinuxusermap.py:249:36: E221 multiple spaces before operator >> ... >> >> Martin > The corner case is fixed now and styling issues corrected as well. > > Tomas Yup, works fine now. ACK. Pushed to master, ipa-3-0. Martin From simo at redhat.com Wed Sep 12 14:29:36 2012 From: simo at redhat.com (Simo Sorce) Date: Wed, 12 Sep 2012 10:29:36 -0400 Subject: [Freeipa-devel] [PATCH] 0078 ipa-client-install: Obtain host TGT from one specific KDC In-Reply-To: <50509675.4010304@redhat.com> References: <5048929E.7080605@redhat.com> <504FA192.5070000@redhat.com> <50506FE3.9070505@redhat.com> <50507B6B.7050908@redhat.com> <505086EA.50402@redhat.com> <50509675.4010304@redhat.com> Message-ID: <1347460176.19861.9.camel@willson.li.ssimo.org> On Wed, 2012-09-12 at 16:04 +0200, Martin Kosek wrote: > On 09/12/2012 02:58 PM, Jan Cholasta wrote: > > Dne 12.9.2012 14:09, Petr Viktorin napsal(a): > >> On 09/12/2012 01:20 PM, Petr Viktorin wrote: > >>> On 09/11/2012 10:39 PM, Rob Crittenden wrote: > >>>> Petr Viktorin wrote: > >>>>> When installing the client, we need to take extra case to only contact > >>>>> the one server we're installing against. Otherwise, in the real world, > >>>>> we might hit a server that hasn't replicated info about the client yet. > >>>>> > >>>>> This patch fixes a bug where kinit attempted to contact a KDC that > >>>>> didn't have the host principal yet. > >>>>> > >>>>> > >>>>> To reproduce: > >>>>> > >>>>> - Install a "master" and "replica" > >>>>> - Change the Kerberos DNS entries to only point to the replica: > >>>>> for REC_NAME in '_kerberos-master._tcp' '_kerberos-master._udp' > >>>>> '_kerberos._tcp' '_kerberos._udp' '_kpasswd._tcp' '_kpasswd._udp'; do > >>>>> ipa dnsrecord-mod $DOMAIN $REC_NAME --srv-rec="0 100 88 > >>>>> $REPLICA_HOSTNAME" > >>>>> done > >>>>> ipa dnsrecord-mod $DOMAIN _ldap._tcp --srv-rec="0 100 389 > >>>>> $MASTER_HOSTNAME" > >>>>> ipa dnsrecord-find $DOMAIN # check > >>>>> - Sever communication between the hosts to disable replication: > >>>>> (on master) > >>>>> iptables -A INPUT -j DROP -p all --source $REPLICA_IP > >>>>> - On client machine, put master as nameserver in /etc/resolv.conf & > >>>>> install client > >>>>> > >>>>> This will fail without the patch. > >>>>> > >>>>> > >>>>> Thanks to Petr Spacek, Simo, and Scott for helping to reproduce and > >>>>> explain the bug. I learned a lot. > >>>>> > >>>>> https://fedorahosted.org/freeipa/ticket/2982 > >>>> > >>>> ACK, pushed to master and ipa-3-0 > >>>> > >>>> rob > >>>> > >>> > >>> The patch broke server installs. Please revert it if you're having > >>> trouble while I look into it. > >>> > >>> > >> > >> I messed up and removed the kinit call entirely when installing on > >> master. Attaching a fix. > >> > > > > Works for me, ACK. > > > > Honza > > > > When the server installation is complete, I was surprised to see I have now > host credentials in my CCACHE: > > # ipa-server-install --setup-dns > ... > ============================================================================== > Setup complete > > Next steps: > 1. You must make sure these network ports are open: > TCP Ports: > * 80, 443: HTTP/HTTPS > * 389, 636: LDAP/LDAPS > * 88, 464: kerberos > * 53: bind > UDP Ports: > * 88, 464: kerberos > * 53: bind > * 123: ntp > > 2. You can now obtain a kerberos ticket using the command: 'kinit admin' > This ticket will allow you to use the IPA tools (e.g., ipa user-add) > and the web user interface. > > Be sure to back up the CA certificate stored in /root/cacert.p12 > This file is required to create replicas. The password for this > file is the Directory Manager password > > # klist > Ticket cache: FILE:/tmp/krb5cc_0 > Default principal: host/vm-086.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM > > Valid starting Expires Service principal > 09/12/12 09:28:24 09/13/12 09:28:24 > krbtgt/IDM.LAB.BOS.REDHAT.COM at IDM.LAB.BOS.REDHAT.COM > 09/12/12 09:28:24 09/13/12 09:28:24 > HTTP/vm-086.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM > 09/12/12 09:28:26 09/13/12 09:28:24 > DNS/vm-086.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM > > > I don't think this is an expected behavior, installer should use a CCACHE > separate from user's default. Definitely, a private install ccache should be used. Please open a ticket. Simo. -- Simo Sorce * Red Hat, Inc * New York From mkosek at redhat.com Wed Sep 12 14:34:22 2012 From: mkosek at redhat.com (Martin Kosek) Date: Wed, 12 Sep 2012 16:34:22 +0200 Subject: [Freeipa-devel] [PATCH] 0078 ipa-client-install: Obtain host TGT from one specific KDC In-Reply-To: <1347460176.19861.9.camel@willson.li.ssimo.org> References: <5048929E.7080605@redhat.com> <504FA192.5070000@redhat.com> <50506FE3.9070505@redhat.com> <50507B6B.7050908@redhat.com> <505086EA.50402@redhat.com> <50509675.4010304@redhat.com> <1347460176.19861.9.camel@willson.li.ssimo.org> Message-ID: <50509D6E.40800@redhat.com> On 09/12/2012 04:29 PM, Simo Sorce wrote: > On Wed, 2012-09-12 at 16:04 +0200, Martin Kosek wrote: >> On 09/12/2012 02:58 PM, Jan Cholasta wrote: >>> Dne 12.9.2012 14:09, Petr Viktorin napsal(a): >>>> On 09/12/2012 01:20 PM, Petr Viktorin wrote: >>>>> On 09/11/2012 10:39 PM, Rob Crittenden wrote: >>>>>> Petr Viktorin wrote: >>>>>>> When installing the client, we need to take extra case to only contact >>>>>>> the one server we're installing against. Otherwise, in the real world, >>>>>>> we might hit a server that hasn't replicated info about the client yet. >>>>>>> >>>>>>> This patch fixes a bug where kinit attempted to contact a KDC that >>>>>>> didn't have the host principal yet. >>>>>>> >>>>>>> >>>>>>> To reproduce: >>>>>>> >>>>>>> - Install a "master" and "replica" >>>>>>> - Change the Kerberos DNS entries to only point to the replica: >>>>>>> for REC_NAME in '_kerberos-master._tcp' '_kerberos-master._udp' >>>>>>> '_kerberos._tcp' '_kerberos._udp' '_kpasswd._tcp' '_kpasswd._udp'; do >>>>>>> ipa dnsrecord-mod $DOMAIN $REC_NAME --srv-rec="0 100 88 >>>>>>> $REPLICA_HOSTNAME" >>>>>>> done >>>>>>> ipa dnsrecord-mod $DOMAIN _ldap._tcp --srv-rec="0 100 389 >>>>>>> $MASTER_HOSTNAME" >>>>>>> ipa dnsrecord-find $DOMAIN # check >>>>>>> - Sever communication between the hosts to disable replication: >>>>>>> (on master) >>>>>>> iptables -A INPUT -j DROP -p all --source $REPLICA_IP >>>>>>> - On client machine, put master as nameserver in /etc/resolv.conf & >>>>>>> install client >>>>>>> >>>>>>> This will fail without the patch. >>>>>>> >>>>>>> >>>>>>> Thanks to Petr Spacek, Simo, and Scott for helping to reproduce and >>>>>>> explain the bug. I learned a lot. >>>>>>> >>>>>>> https://fedorahosted.org/freeipa/ticket/2982 >>>>>> >>>>>> ACK, pushed to master and ipa-3-0 >>>>>> >>>>>> rob >>>>>> >>>>> >>>>> The patch broke server installs. Please revert it if you're having >>>>> trouble while I look into it. >>>>> >>>>> >>>> >>>> I messed up and removed the kinit call entirely when installing on >>>> master. Attaching a fix. >>>> >>> >>> Works for me, ACK. >>> >>> Honza >>> >> >> When the server installation is complete, I was surprised to see I have now >> host credentials in my CCACHE: >> >> # ipa-server-install --setup-dns >> ... >> ============================================================================== >> Setup complete >> >> Next steps: >> 1. You must make sure these network ports are open: >> TCP Ports: >> * 80, 443: HTTP/HTTPS >> * 389, 636: LDAP/LDAPS >> * 88, 464: kerberos >> * 53: bind >> UDP Ports: >> * 88, 464: kerberos >> * 53: bind >> * 123: ntp >> >> 2. You can now obtain a kerberos ticket using the command: 'kinit admin' >> This ticket will allow you to use the IPA tools (e.g., ipa user-add) >> and the web user interface. >> >> Be sure to back up the CA certificate stored in /root/cacert.p12 >> This file is required to create replicas. The password for this >> file is the Directory Manager password >> >> # klist >> Ticket cache: FILE:/tmp/krb5cc_0 >> Default principal: host/vm-086.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM >> >> Valid starting Expires Service principal >> 09/12/12 09:28:24 09/13/12 09:28:24 >> krbtgt/IDM.LAB.BOS.REDHAT.COM at IDM.LAB.BOS.REDHAT.COM >> 09/12/12 09:28:24 09/13/12 09:28:24 >> HTTP/vm-086.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM >> 09/12/12 09:28:26 09/13/12 09:28:24 >> DNS/vm-086.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM >> >> >> I don't think this is an expected behavior, installer should use a CCACHE >> separate from user's default. > > Definitely, > a private install ccache should be used. > Please open a ticket. > > Simo. > This is caused by a patch pushed today (in a scope of a fix for ticket 2982). Petr Viktorin is working on a fix which will be sent soon, so I think that ticket is not necessary in this case. Martin From mkosek at redhat.com Wed Sep 12 15:29:44 2012 From: mkosek at redhat.com (Martin Kosek) Date: Wed, 12 Sep 2012 17:29:44 +0200 Subject: [Freeipa-devel] [PATCH 0006] Improves sssd.conf handling during ipa-client uninstall In-Reply-To: <503E1114.3080309@redhat.com> References: <717623750.1969105.1345208663118.JavaMail.root@redhat.com> <503B5D09.7040401@redhat.com> <503B6EC8.3060508@redhat.com> <20120827133727.GG8612@zeppelin.brq.redhat.com> <503B8A65.30100@redhat.com> <503E1114.3080309@redhat.com> Message-ID: <5050AA68.2010300@redhat.com> On 08/29/2012 02:54 PM, Tomas Babej wrote: > On 08/27/2012 04:55 PM, Martin Kosek wrote: >> On 08/27/2012 03:37 PM, Jakub Hrozek wrote: >>> On Mon, Aug 27, 2012 at 02:57:44PM +0200, Martin Kosek wrote: >>>> I think that the right behavior of SSSD conf uninstall should be the >>>> following: >>>> >>>> * sssd.conf existed before IPA install + non-IPA domains in sssd.conf found: >>>> - move backed conf up sssd.conf.bkp (and inform the user) >>>> - use SSSDConfig delete_domain function to remove ipa domain from sssd.conf >>>> - restart sssd afterwards >>> I'm confused here, which of the files is the original >>> pre-ipa-client-install file? >> This is the "backed up sssd.conf". I thought that it may be useful for user to >> still have an access to it after uninstall. >> >>> How does the non-ipa domain end up in the sssd.conf file? Does it have >>> to be configured manually or does ipa-client-install merge the list of >>> domains on installation? >> ipa-client-install merge the list of the domains. It overrides the old >> sssd.conf only when we cannot parse the sssd.conf and --preserve-sssd option >> was not set. >> >> Martin > Hi, > > The sssd.conf file is no longer left behind in case sssd was not > configured before the installation. However, the patch goes behind > the scope of this ticked and improves the handling of sssd.conf > during the ipa-client-install --uninstall in general. > > The current behaviour (well documented in source code) is as follows: > - In general, the IPA domain is simply removed from the sssd.conf > file, instead of sssd.conf being rewritten from the backup. This > preserves any domains added after installation. > > - If sssd.conf existed before the installation, it is restored to > sssd.conf.bkp. However, any IPA domains from pre-installation > sssd.conf should have been merged during the installation. > > - If sssd.conf did not exist before the installation, and no other > domains than IPA domain exist in it, the patch makes sure that > sssd.conf is moved to sssd.conf.deleted so user experiences no > crash during any next installation due to its existence. > > https://fedorahosted.org/freeipa/ticket/2740 > > Tomas > Good job, SSSD uninstall process now looks more consistent and better documented. I just found the following (mainly minor) issues. Comments in the patch: diff --git a/ipa-client/ipa-install/ipa-client-install b/ipa-client/ipa-install/ipa-client-install index 2e65921e8de2dfe68443f5b5875954d71dd48ed2..c5cef15e1fb3a3e1d7cfd070f4288d3839accfc8 100755 --- a/ipa-client/ipa-install/ipa-client-install +++ b/ipa-client/ipa-install/ipa-client-install @@ -183,6 +183,36 @@ def nssldap_exists(): return (retval, files_found) +# helper function for uninstall +# deletes IPA domain from sssd.conf +def delete_IPA_domain(): Function names should be lowercase -> delete_ipa_domain + sssd = ipaservices.service('sssd') + try: + sssdconfig = SSSDConfig.SSSDConfig() + sssdconfig.import_config() + domains = sssdconfig.list_active_domains() + + IPA_domain_name = None Variables should be lowercase -> ipa_domain_name + + for name in domains: + domain = sssdconfig.get_domain(name) + try: + provider = domain.get_option('id_provider') + if provider == "ipa": + IPA_domain_name = name + break + except SSSDConfig.NoOptionError: + continue + + if IPA_domain_name != None: Do not use != with None, True, False - use "is not None". + sssdconfig.delete_domain(IPA_domain_name) + sssdconfig.write() + else: + root_logger.warning("IPA domain could not be found in " + + "sssd.conf and therefore not deleted") + except IOError: + root_logger.warning("IPA domain could not be deleted. No access to the sssd.conf file.") There should be full path to sssd.conf in this error message. It is very useful sometimes. + def uninstall(options, env): if not fstore.has_files(): @@ -212,7 +242,12 @@ def uninstall(options, env): sssdconfig = SSSDConfig.SSSDConfig() sssdconfig.import_config() domains = sssdconfig.list_active_domains() - if len(domains) > 1: + all_domains = sssdconfig.list_domains() + + # we consider all the domains, because handling sssd.conf + # during uninstall is dependant on was_sssd_configured flag + # so the user does not lose info about inactive domains + if len(all_domains) > 1: # There was more than IPA domain configured was_sssd_configured = True for name in domains: @@ -349,6 +384,62 @@ def uninstall(options, env): "Failed to remove krb5/LDAP configuration: %s", str(e)) return CLIENT_INSTALL_ERROR + # Next if-elif-elif construction deals with sssd.conf file. + # Old pre-IPA domains are preserved due merging the old sssd.conf + # during the installation of ipa-client but any new domains are + # only present in sssd.conf now, so we don't want to delete them + # by rewriting sssd.conf file. IPA domain is removed gracefully. + + # SSSD was installed before our installation and other non-IPA domains + # found, restore backed up sssd.conf to sssd.conf.bkp and remove IPA + # domain from the current sssd.conf + if was_sssd_installed and was_sssd_configured: + root_logger.info( + "The original configuration of SSSD included other domains than " + + "the IPA-based one.") + + delete_IPA_domain() + + try: + os.rename("/etc/sssd/sssd.conf","/etc/sssd/sssd.conf.current") + if fstore.restore_file("/etc/sssd/sssd.conf"): + os.rename("/etc/sssd/sssd.conf","/etc/sssd/sssd.conf.bkp") + os.rename("/etc/sssd/sssd.conf.current","/etc/sssd/sssd.conf") I don't like this very much. I would rather not mess with /etc/sssd/sssd.conf, it may surprise other tools/user and we could also end with no /etc/sssd/sssd.conf at all if OSError is raised in second or third step. I would rather enhance fstore and implement function like "restore_file_to" which would accept a second parameter with a custom path, like "/etc/sssd/sssd.conf.bkp". + except OSError: + pass The exception should be at least logged to debug log. + + root_logger.info("Original pre-IPA SSSD configuration file was " + + "restored to /etc/sssd/sssd.conf.bkp.") + root_logger.info("IPA domain removed from current one, " + + "restarting SSSD service") + sssd = ipaservices.service('sssd') + try: + sssd.restart() + except CalledProcessError: + root_logger.warning("SSSD service restart was unsuccessful.") + + # SSSD was not installed before our installation, but other domains found, + # delete IPA domain, but leave other domains intact + elif not was_sssd_installed and was_sssd_configured: + delete_IPA_domain() + root_logger.info("Other domains than IPA domain found, " + + "IPA domain was removed from sssd.conf.") + try: + sssd.restart() + except CalledProcessError: + root_logger.warning("SSSD service restart was unsuccessful.") + + # SSSD was not installed before our installation, and no other domains + # than IPA are configured in sssd.conf - make sure config file is removed + elif not was_sssd_installed and not was_sssd_configured: + try: + os.rename("/etc/sssd/sssd.conf","/etc/sssd/sssd.conf.deleted") + except OSError: + pass Error should be logged to debug log. + + root_logger.info("Redundant SSSD configuration file " + + "/etc/sssd/sssd.conf was moved to /etc/sssd.conf.deleted") + if fstore.has_files(): root_logger.info("Restoring client configuration files") fstore.restore_all_files() @@ -428,20 +519,6 @@ def uninstall(options, env): if was_sshd_configured and ipaservices.knownservices.sshd.is_running(): ipaservices.knownservices.sshd.restart() - if was_sssd_installed and was_sssd_configured: - # SSSD was installed before our installation, config now is restored, restart it - root_logger.info( - "The original configuration of SSSD included other domains than " + - "the IPA-based one.") - root_logger.info( - "Original configuration file was restored, restarting SSSD " + - "service.") - sssd = ipaservices.service('sssd') - try: - sssd.restart() - except CalledProcessError: - root_logger.warning("SSSD service restart was unsuccessful.") - if not options.unattended: root_logger.info( "The original nsswitch.conf configuration has been restored.") -- 1.7.11.4 Martin From pviktori at redhat.com Wed Sep 12 16:01:49 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Wed, 12 Sep 2012 18:01:49 +0200 Subject: [Freeipa-devel] [PATCH] 0077 Check direct/reverse hostname/address resolution in ipa-replica-install In-Reply-To: <504FA7A3.5080201@redhat.com> References: <5046009E.2020306@redhat.com> <50463DE6.9070300@redhat.com> <5046FF0E.1000504@redhat.com> <504FA7A3.5080201@redhat.com> Message-ID: <5050B1ED.70307@redhat.com> On 09/11/2012 11:05 PM, Rob Crittenden wrote: > Petr Viktorin wrote: >> On 09/04/2012 07:44 PM, Rob Crittenden wrote: >>> Petr Viktorin wrote: >>>> >>>> https://fedorahosted.org/freeipa/ticket/2845 >>> >>> Shouldn't this also call verify_fqdn() on the local hostname and not >>> just the master? I think this would eventually fail in the conncheck but >>> what if that was skipped? >>> >>> rob >> >> A few lines above there is a call to get_host_name, which will call >> verify_fqdn. >> > > I double-checked this, it fails in conncheck. Here are my steps: > > # ipa-server-install --setup-dns > # ipa-replica-prepare replica.example.com --ip-address=192.168.100.2 > # ipa host-del replica.example.com > > On replica, set DNS to IPA master, with hostname in /etc/hosts. > > # ipa-replica-install ... > > The verify_fqdn() passes because the resolver uses /etc/hosts. > > The conncheck fails: > > Execute check on remote master > Check connection from master to remote replica 'replica.example.com': > > Remote master check failed with following error message(s): > Could not chdir to home directory /home/admin: No such file or directory > Port check failed! Unable to resolve host name 'replica.example.com' > > Connection check failed! > Please fix your network settings according to error messages above. > If the check results are not valid it can be skipped with > --skip-conncheck parameter. > > The DNS test happens much further after this, and I get why, I just > don't see how useful it is unless the --skip-conncheck is used. For the record, it's because we need to check if the host has DNS installed. We need a LDAP connection to check this. > ipa-replica-install ~rcrit/replica-info-replica.example.com.gpg > --skip-conncheck > Directory Manager (existing master) password: > > ipa : ERROR Could not resolve hostname replica.example.com > using DNS. Clients may not function properly. Please check your DNS > setup. (Note that this check queries IPA DNS directly and ignores > /etc/hosts.) > Continue? [no]: > > So I guess, what are the intentions here? It is certainly better than > before. > > rob If the replica is in the master's /etc/hosts, but not in DNS, the conncheck will succeed. This check explicitly queries IPA records only and ignores /etc/hosts so it'll notice this case and warn. -- Petr? From pviktori at redhat.com Wed Sep 12 16:02:12 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Wed, 12 Sep 2012 18:02:12 +0200 Subject: [Freeipa-devel] [PATCH] 0078 ipa-client-install: Obtain host TGT from one specific KDC In-Reply-To: <50509675.4010304@redhat.com> References: <5048929E.7080605@redhat.com> <504FA192.5070000@redhat.com> <50506FE3.9070505@redhat.com> <50507B6B.7050908@redhat.com> <505086EA.50402@redhat.com> <50509675.4010304@redhat.com> Message-ID: <5050B204.4010408@redhat.com> On 09/12/2012 04:04 PM, Martin Kosek wrote: > On 09/12/2012 02:58 PM, Jan Cholasta wrote: >> Dne 12.9.2012 14:09, Petr Viktorin napsal(a): >>> On 09/12/2012 01:20 PM, Petr Viktorin wrote: >>>> On 09/11/2012 10:39 PM, Rob Crittenden wrote: >>>>> Petr Viktorin wrote: >>>>>> When installing the client, we need to take extra case to only contact >>>>>> the one server we're installing against. Otherwise, in the real world, >>>>>> we might hit a server that hasn't replicated info about the client yet. >>>>>> >>>>>> This patch fixes a bug where kinit attempted to contact a KDC that >>>>>> didn't have the host principal yet. >>>>>> >>>>>> >>>>>> To reproduce: >>>>>> >>>>>> - Install a "master" and "replica" >>>>>> - Change the Kerberos DNS entries to only point to the replica: >>>>>> for REC_NAME in '_kerberos-master._tcp' '_kerberos-master._udp' >>>>>> '_kerberos._tcp' '_kerberos._udp' '_kpasswd._tcp' '_kpasswd._udp'; do >>>>>> ipa dnsrecord-mod $DOMAIN $REC_NAME --srv-rec="0 100 88 >>>>>> $REPLICA_HOSTNAME" >>>>>> done >>>>>> ipa dnsrecord-mod $DOMAIN _ldap._tcp --srv-rec="0 100 389 >>>>>> $MASTER_HOSTNAME" >>>>>> ipa dnsrecord-find $DOMAIN # check >>>>>> - Sever communication between the hosts to disable replication: >>>>>> (on master) >>>>>> iptables -A INPUT -j DROP -p all --source $REPLICA_IP >>>>>> - On client machine, put master as nameserver in /etc/resolv.conf & >>>>>> install client >>>>>> >>>>>> This will fail without the patch. >>>>>> >>>>>> >>>>>> Thanks to Petr Spacek, Simo, and Scott for helping to reproduce and >>>>>> explain the bug. I learned a lot. >>>>>> >>>>>> https://fedorahosted.org/freeipa/ticket/2982 >>>>> >>>>> ACK, pushed to master and ipa-3-0 >>>>> >>>>> rob >>>>> >>>> >>>> The patch broke server installs. Please revert it if you're having >>>> trouble while I look into it. >>>> >>>> >>> >>> I messed up and removed the kinit call entirely when installing on >>> master. Attaching a fix. >>> >> >> Works for me, ACK. >> >> Honza >> > > When the server installation is complete, I was surprised to see I have now > host credentials in my CCACHE: > > # ipa-server-install --setup-dns > ... > ============================================================================== > Setup complete > > Next steps: > 1. You must make sure these network ports are open: > TCP Ports: > * 80, 443: HTTP/HTTPS > * 389, 636: LDAP/LDAPS > * 88, 464: kerberos > * 53: bind > UDP Ports: > * 88, 464: kerberos > * 53: bind > * 123: ntp > > 2. You can now obtain a kerberos ticket using the command: 'kinit admin' > This ticket will allow you to use the IPA tools (e.g., ipa user-add) > and the web user interface. > > Be sure to back up the CA certificate stored in /root/cacert.p12 > This file is required to create replicas. The password for this > file is the Directory Manager password > > # klist > Ticket cache: FILE:/tmp/krb5cc_0 > Default principal: host/vm-086.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM > > Valid starting Expires Service principal > 09/12/12 09:28:24 09/13/12 09:28:24 > krbtgt/IDM.LAB.BOS.REDHAT.COM at IDM.LAB.BOS.REDHAT.COM > 09/12/12 09:28:24 09/13/12 09:28:24 > HTTP/vm-086.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM > 09/12/12 09:28:26 09/13/12 09:28:24 > DNS/vm-086.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM > > > I don't think this is an expected behavior, installer should use a CCACHE > separate from user's default. > > Martin I need to slow down. Thanks for the catch. Attaching another fix. -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0080-02-Add-host-key-to-temporary-key-cache-on-server-instal.patch Type: text/x-patch Size: 1281 bytes Desc: not available URL: From pviktori at redhat.com Wed Sep 12 16:40:52 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Wed, 12 Sep 2012 18:40:52 +0200 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <504F86C0.6060907@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> <1347329065.24731.101.camel@aleeredhat.laptop> <504F318F.3030603@redhat.com> <504F35AD.3030905@redhat.com> <1347373691.24731.125.camel@aleeredhat.laptop> <504F4CE0.9080706@redhat.com> <504F67C5.4000600@redhat.com> <504F86C0.6060907@redhat.com> Message-ID: <5050BB14.3050502@redhat.com> A new Dogtag build with changed pkispawn/pkidestroy locations should be out later today. The attached patch should work with that build. On 09/11/2012 08:45 PM, Rob Crittenden wrote: > Petr Viktorin wrote: >> On 09/11/2012 04:38 PM, Rob Crittenden wrote: >>> Ade Lee wrote: >>>> On Tue, 2012-09-11 at 08:59 -0400, Rob Crittenden wrote: >>>>> Petr Viktorin wrote: >>>>>> On 09/11/2012 04:04 AM, Ade Lee wrote: >>>>>>> On Mon, 2012-09-10 at 16:58 -0400, Rob Crittenden wrote: >>>>>>>> Petr Viktorin wrote: >>>>>>>>> Attaching rebased and squashed patches. I've done some testing >>>>>>>>> with >>>>>>>>> them >>>>>>>>> but please test some more. >>>>>>>>> >>>>>>>> >>>>>>>> Most of these aren't IPA issues, but dogtag issues. I'll try to >>>>>>>> split >>>>>>>> them out. >>>>>>>> >>>>>>>> IPA: >>>>>>>> >>>>>>>> For the configuration files in install/conf to be updated at rpm >>>>>>>> update >>>>>>>> time the VERSION needs to be incremented. >>>>>> >>>>>> These files should stay the same since on upgrade we're still using a >>>>>> Dogtag 9 style instance. The Dogtag 10 ports are only used in new >>>>>> installs. >>>>>> >>>>>>>> The ipa package lacks any updated dogtag dependencies, so I abused >>>>>>>> it. >>>>>> >>>>>> What should the updated dependencies be? Since it should work with >>>>>> both >>>>>> dogtag 9 and 10, I don't see how they should change. >>>>> >>>>> I don't know either, but we need to prevent people from installing >>>>> incompatible package combinations. >>>>> >>>> Would'nt the Conflicts: ipa < 3.0 in pki-ca mentioned below satisfy >>>> this >>>> requirement? The main concern is that you must have ipa 3.0 if you >>>> have >>>> dogtag 10. >>>> >>>> Given that dogtag is consumed by IPA though, it makes more sense to put >>>> the relevant conflicts in IPA rather than in dogtag. So in this case, >>>> that would mean putting Conflicts: pki-ca >= 10.0 in IPA 2.x. >>>> Recall that dogtag 10 will only be officially available in f18+. >>> >>> That isn't enough. If a F-17 user with IPA 2.2 installed upgrades to >>> F-18 they would be able to install dogtag 10 and blow up their IPA >>> server. >>> >>>> >>>>>>>> I installed IPA with dogtag 9 and created a replica. >>>>>>>> >>>>>>>> I updated the IPA bits, that worked fine. >>>>>>>> >>>>>>>> I updated to dogtag 10 and now the CA doesn't work on the master, >>>>>>>> including starting the dogtag instance. Note that the rpm update >>>>>>>> process >>>>>>>> worked, no notice that the CA service didn't restart. >>>>>>>> >>>>>>> Did you try to restart the CA with selinux in permissive mode? >>>>>>> This is >>>>>>> still required right now until I get the selinux policy all >>>>>>> straightened >>>>>>> out. >>>>>>> >>>>>>> There is also a separate dogtag ticket (which is currently being >>>>>>> worked >>>>>>> on) to restart existing dogtag instances when dogtag is upgraded >>>>>>> from >>>>>>> 9->10. >>>>>> >>>>>> In permissive mode, this upgrade works for me. >>>>> >>>>> I was in enforcing mode but saw no AVCs. What is the ETA on fixing >>>>> this? >>>>> >>>> >>>> Within the next week or two, I need to finish the IPA merge database >>>> patch first, and then co-ordinate changes with the selinux guys. >>>> >>>>>> >>>>>> >>>>>> Sometimes I do get this error intermittently: >>>>>> >>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to >>>>>> communicate with CMS (Service Temporarily Unavailable) >>>>>> >>>>>> Usually, waiting a couple of minutes clears this up. Perhaps we are >>>>>> not >>>>>> doing startup detection correctly. Ade mentioned that waiting for >>>>>> ports >>>>>> may not be ideal. How do we know if Dogtag is initialized? >>>>> >>>>> Years ago we had discussed with Andrew and Matt creating a URI that >>>>> can >>>>> be queried to determine dogtag status. I don't think that ever went >>>>> anywhere. >>>>> >>>> Petr, this happens only on reboot, right? And not on regular "service >>>> ipa restart"? >> >> I've now seen it happen right after a 9 ? 10 upgrade. >> >>>> Yeah, I remember this conversation - and even created a bug for it at >>>> some point. This went away because the mechanism you were using seemed >>>> to be working. The timing may be very different now with tomcat 7 and >>>> systemd. I'll open a dogtag trac ticket for this. >>> >>> Ok. >>> >>>> >>>>>> >>>>>>>> Uninstalling failed because it tried to run pkidestroy and not >>>>>>>> pkiremove. >>>>>> >>>>>> I was under the impression that pkidestroy was the correct command to >>>>>> remove an upgraded instance. I'll check with Ade. >>>>>> >>>>>>> I'll test this too. >>>>>>> >>>>>>>> The contents of the file passed to pkispawn should be logged so we >>>>>>>> can >>>>>>>> see exactly what was passed in. >>>>>>>> >>>>>>> Its a pretty big file. You might want to only log the >>>>>>> modifications. >>>>>>> Or save the file somewhere. >>>>>> >>>>>> Our logs are pretty verbose, so that shouldn't be a problem. I'll >>>>>> put it >>>>>> in the next version of the patch. >>>>> >>>>> The question to ask is: would you need the contents of this file if >>>>> all >>>>> you got were logs and needed to evaluate why installation failed? In >>>>> most cases this is yes. >>>>> >>>> Up to you guys. There is a patch I am working on in which I will be >>>> logging the object that is being passed to the server from pkispawn. >>>> That - and the diffs to the standard config file as I mentioned above - >>>> will likely be sufficient to debug almost all cases. >>>> >>>> Make sure not to log any passwords. >>>> >> >> Thanks for the catch. Attaching updated patch that sanitizes the >> passwords. >> >>>>>>>> DOGTAG: >>>>>>>> >>>>>>>> When upgrading using the dogtag-devel repo I had to specify >>>>>>>> pki-tools.x86_64 otherwise it tried to install both 32 and 64-bit >>>>>>>> versions (and failed). >>>>>>>> >>>>>>>> I ended up running: yum update pki-ca tomcatjss pki-tools.x86_64 >>>>>>>> --enablerepo=dogtag-devel --enablerepo=updates-testing >>>>>>>> >>>>>>> We'll look into this. I think I know why this happens. >>>>>>> >>>>>>>> What happens if someone manually upgrades pki-ca without first >>>>>>>> updating >>>>>>>> ipa? I think that pki-ca is going to need a Conflicts ipa < 3.0 in >>>>>>>> it. >>>>>>> >>>>>>> We can add that. >>>>>>> >>>>>>>> certificate renewal failed. I spent far too long trying to figure >>>>>>>> out >>>>>>>> why tomcat wasn't listening on port 9180 but failed. I think >>>>>>>> 9180 is >>>>>>>> actually the old server, right? So another missing dependency on a >>>>>>>> fixed >>>>>>>> certmonger? >>>>>>>> >>>>>>>> The best I could find was the certmonger error: >>>>>>>> >>>>>>>> ca-error: Error 7 connecting to >>>>>>>> http://edsel.example.com:9180/ca/ee/ca/profileSubmit: Couldn't >>>>>>>> connect >>>>>>>> to server. >>>>>>>> >>>>>> >>>>>> I'll set the certmonger min version to 0.60 as per Nalin's mail. >>>>> >>>>> Ok. >>>>> >>>>>>> Is this cert renewal on a dogtag 10 instance? Or the upgraded >>>>>>> dogtag 9? >>>>>>> If its dogtag 10, perhaps you do not have the certmonger version >>>>>>> that >>>>>>> has the relevant fix? If its dogtag 9, then we need to figure out >>>>>>> whats >>>>>>> going on. That reminds me - I need to file a bug to allow >>>>>>> certmonger to >>>>>>> talk to the newly defined dogtag ports. Do you have selinux >>>>>>> permissive? >>>>>>> >>>>>>>> There is no man page for pkispawn/pkidestroy :-( According to the >>>>>>>> FHS >>>>>>>> these should not be in /bin but in /usr/sbin (not end-user >>>>>>>> commands). >>>>>>>> >>>>>>> There is a trac ticket open for man pages for pkispawn and >>>>>>> pkidestroy. >>>>>>> We plan to complete this ticket by the time f18 is released. >>>>>>> >>>>>>> We'll look into the location of pkispawn/pkicreate. >>>>>>> >>>>>>>> The output of pkicreate/pkisilent was really terrible and not >>>>>>>> usable at >>>>>>>> all so we didn't display it when failures occurred. It looks like >>>>>>>> that >>>>>>>> has been addressed, at least for the case where a CA is already >>>>>>>> configured and you try to install IPA. Perhaps we should capture >>>>>>>> stderr >>>>>>>> and display that instead of the command-line of pkispawn? Again, a >>>>>>>> man >>>>>>>> page would help with the integration. >>>>>>>> >>>>>>>> 2012-09-10T20:51:45Z DEBUG [2/18]: configuring certificate server >>>>>>>> instance >>>>>>>> 2012-09-10T20:51:45Z DEBUG args=/bin/pkispawn -s CA -f >>>>>>>> /tmp/tmp_Urraq >>>>>>>> 2012-09-10T20:51:45Z DEBUG stdout= >>>>>>>> 2012-09-10T20:51:45Z DEBUG stderr=pkispawn : ERROR ....... >>>>>>>> PKI >>>>>>>> subsystem 'CA' for instance 'pki-tomcat' already exists! >>>>>>>> >>>>>>>> 2012-09-10T20:51:45Z CRITICAL failed to configure ca instance >>>>>>>> Command >>>>>>>> '/bin/pkispawn -s CA -f /tmp/tmp_Urraq' returned non-zero exit >>>>>>>> status 1 >>>>>>>> >>>>>>> That may be a good idea. Of course. thats an IPA thing, right? >>>>> >>>>> Right, not a show-stopper for this but a new ticket should be >>>>> opened to >>>>> track it. >> >> https://fedorahosted.org/freeipa/ticket/3072 > > Thanks. > > I tested a 2.2 -> 3.0 upgrade with dogtag 9 and it seems to have failed > to restart something: > > [ post rpm -Uvh dist/rpms/freeipa*.rpm ] > > [root at edsel freeipa]# ipa cert-show 1 > ipa: ERROR: Certificate operation cannot be completed: Unable to > communicate with CMS (Service Temporarily Unavailable) I'm getting this error as well. systemctl shows that pki-cad at pki-ca.service as failed. But, after a ipactl restart, the service starts correctly and `ipa cert-show` works again (in permissive mode). I didn't have time to finish investigating this today but I'll work on it more tomorrow. > [root at edsel freeipa]# ipactl restart > Restarting Directory Service > Restarting KDC Service > Restarting KPASSWD Service > Restarting MEMCACHE Service > Restarting HTTP Service > Restarting CA Service > [root at edsel freeipa]# ipa cert-show 1 > Certificate: MIIDjTCCAnWgAwIBAgIBATANBgk > ... > > The apache log had this: > > [Tue Sep 11 14:35:42 2012] [error] (111)Connection refused: proxy: AJP: > attempt to connect to 127.0.0.1:9447 (localhost) failed > [Tue Sep 11 14:35:42 2012] [error] ap_proxy_connect_backend disabling > worker for (localhost) > [Tue Sep 11 14:35:42 2012] [error] proxy: AJP: failed to make connection > to backend: localhost > > So I'm gathering that dogtag didn't restart properly, but I'm just > guessing. It could have been the PKI-IPA ds instance too, I'm not sure > where to check. > > I also noticed this in /var/log/ipaupgrade.log: > > 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl start certmonger.service > 2012-09-11T18:28:22Z DEBUG stdout= > 2012-09-11T18:28:22Z DEBUG stderr= > 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl is-active certmonger.service > 2012-09-11T18:28:22Z DEBUG stdout=active > ... > ... [ snip certutil output ] > ... > 2012-09-11T18:28:52Z DEBUG args=/usr/bin/getcert start-tracking -d > /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c > dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert > "auditSigningCert cert-pki-ca" -P XXXXXXXX > 2012-09-11T18:28:52Z DEBUG stdout=Please verify that the certmonger > service is still running. > > 2012-09-11T18:28:52Z DEBUG stderr= > 2012-09-11T18:28:52Z INFO File > "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", > line 614, in run_script > return_value = main_function() > > File "/usr/sbin/ipa-upgradeconfig", line 540, in main > enable_certificate_renewal(api.env.realm) > > File "/usr/sbin/ipa-upgradeconfig", line 455, in > enable_certificate_renewal > ca.configure_renewal() > > File > "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line > 1298, in configure_renewal > self.dogtag_constants.ALIAS_DIR, 'renew_ca_cert "%s"' % nickname) > > File "/usr/lib/python2.7/site-packages/ipapython/certmonger.py", line > 394, in dogtag_start_tracking > (stdout, stderr, returncode) = ipautil.run(args, nolog=[pin]) > > File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line > 309, in run > raise CalledProcessError(p.returncode, args) > > 2012-09-11T18:28:52Z INFO The ipa-upgradeconfig command failed, > exception: CalledProcessError: Command '/usr/bin/getcert start-tracking > -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c > dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert > "auditSigningCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit status 1 > > I'm not sure if this is related to this patch or not. If it isn't can > you file a new ticket on it, or add it the 2.2 -> 3.0 upgrade ticket? I've opened a ticket. -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0074-07-Use-Dogtag-10-only-when-it-is-available.patch Type: text/x-patch Size: 59913 bytes Desc: not available URL: From pviktori at redhat.com Wed Sep 12 16:43:55 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Wed, 12 Sep 2012 18:43:55 +0200 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <504F9333.8040707@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> <1347329065.24731.101.camel@aleeredhat.laptop> <504F318F.3030603@redhat.com> <504F35AD.3030905@redhat.com> <1347373691.24731.125.camel@aleeredhat.laptop> <504F4CE0.9080706@redhat.com> <504F67C5.4000600@redhat.com> <504F86C0.6060907@redhat.com> <504F8F8B.7040709@redhat.com> <504F9333.8040707@redhat.com> Message-ID: <5050BBCB.1020500@redhat.com> On 09/11/2012 09:38 PM, Rob Crittenden wrote: > Rob Crittenden wrote: >> Rob Crittenden wrote: >>> Petr Viktorin wrote: >>>> On 09/11/2012 04:38 PM, Rob Crittenden wrote: >>>>> Ade Lee wrote: >>>>>> On Tue, 2012-09-11 at 08:59 -0400, Rob Crittenden wrote: >>>>>>> Petr Viktorin wrote: >>>>>>>> On 09/11/2012 04:04 AM, Ade Lee wrote: >>>>>>>>> On Mon, 2012-09-10 at 16:58 -0400, Rob Crittenden wrote: >>>>>>>>>> Petr Viktorin wrote: >>>>>>>>>>> Attaching rebased and squashed patches. I've done some testing >>>>>>>>>>> with >>>>>>>>>>> them >>>>>>>>>>> but please test some more. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Most of these aren't IPA issues, but dogtag issues. I'll try to >>>>>>>>>> split >>>>>>>>>> them out. >>>>>>>>>> >>>>>>>>>> IPA: >>>>>>>>>> >>>>>>>>>> For the configuration files in install/conf to be updated at rpm >>>>>>>>>> update >>>>>>>>>> time the VERSION needs to be incremented. >>>>>>>> >>>>>>>> These files should stay the same since on upgrade we're still >>>>>>>> using a >>>>>>>> Dogtag 9 style instance. The Dogtag 10 ports are only used in new >>>>>>>> installs. >>>>>>>> >>>>>>>>>> The ipa package lacks any updated dogtag dependencies, so I >>>>>>>>>> abused >>>>>>>>>> it. >>>>>>>> >>>>>>>> What should the updated dependencies be? Since it should work with >>>>>>>> both >>>>>>>> dogtag 9 and 10, I don't see how they should change. >>>>>>> >>>>>>> I don't know either, but we need to prevent people from installing >>>>>>> incompatible package combinations. >>>>>>> >>>>>> Would'nt the Conflicts: ipa < 3.0 in pki-ca mentioned below satisfy >>>>>> this >>>>>> requirement? The main concern is that you must have ipa 3.0 if you >>>>>> have >>>>>> dogtag 10. >>>>>> >>>>>> Given that dogtag is consumed by IPA though, it makes more sense to >>>>>> put >>>>>> the relevant conflicts in IPA rather than in dogtag. So in this >>>>>> case, >>>>>> that would mean putting Conflicts: pki-ca >= 10.0 in IPA 2.x. >>>>>> Recall that dogtag 10 will only be officially available in f18+. >>>>> >>>>> That isn't enough. If a F-17 user with IPA 2.2 installed upgrades to >>>>> F-18 they would be able to install dogtag 10 and blow up their IPA >>>>> server. >>>>> >>>>>> >>>>>>>>>> I installed IPA with dogtag 9 and created a replica. >>>>>>>>>> >>>>>>>>>> I updated the IPA bits, that worked fine. >>>>>>>>>> >>>>>>>>>> I updated to dogtag 10 and now the CA doesn't work on the master, >>>>>>>>>> including starting the dogtag instance. Note that the rpm update >>>>>>>>>> process >>>>>>>>>> worked, no notice that the CA service didn't restart. >>>>>>>>>> >>>>>>>>> Did you try to restart the CA with selinux in permissive mode? >>>>>>>>> This is >>>>>>>>> still required right now until I get the selinux policy all >>>>>>>>> straightened >>>>>>>>> out. >>>>>>>>> >>>>>>>>> There is also a separate dogtag ticket (which is currently being >>>>>>>>> worked >>>>>>>>> on) to restart existing dogtag instances when dogtag is upgraded >>>>>>>>> from >>>>>>>>> 9->10. >>>>>>>> >>>>>>>> In permissive mode, this upgrade works for me. >>>>>>> >>>>>>> I was in enforcing mode but saw no AVCs. What is the ETA on fixing >>>>>>> this? >>>>>>> >>>>>> >>>>>> Within the next week or two, I need to finish the IPA merge database >>>>>> patch first, and then co-ordinate changes with the selinux guys. >>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Sometimes I do get this error intermittently: >>>>>>>> >>>>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to >>>>>>>> communicate with CMS (Service Temporarily Unavailable) >>>>>>>> >>>>>>>> Usually, waiting a couple of minutes clears this up. Perhaps we are >>>>>>>> not >>>>>>>> doing startup detection correctly. Ade mentioned that waiting for >>>>>>>> ports >>>>>>>> may not be ideal. How do we know if Dogtag is initialized? >>>>>>> >>>>>>> Years ago we had discussed with Andrew and Matt creating a URI that >>>>>>> can >>>>>>> be queried to determine dogtag status. I don't think that ever went >>>>>>> anywhere. >>>>>>> >>>>>> Petr, this happens only on reboot, right? And not on regular >>>>>> "service >>>>>> ipa restart"? >>>> >>>> I've now seen it happen right after a 9 ? 10 upgrade. >>>> >>>>>> Yeah, I remember this conversation - and even created a bug for it at >>>>>> some point. This went away because the mechanism you were using >>>>>> seemed >>>>>> to be working. The timing may be very different now with tomcat 7 >>>>>> and >>>>>> systemd. I'll open a dogtag trac ticket for this. >>>>> >>>>> Ok. >>>>> >>>>>> >>>>>>>> >>>>>>>>>> Uninstalling failed because it tried to run pkidestroy and not >>>>>>>>>> pkiremove. >>>>>>>> >>>>>>>> I was under the impression that pkidestroy was the correct >>>>>>>> command to >>>>>>>> remove an upgraded instance. I'll check with Ade. >>>>>>>> >>>>>>>>> I'll test this too. >>>>>>>>> >>>>>>>>>> The contents of the file passed to pkispawn should be logged >>>>>>>>>> so we >>>>>>>>>> can >>>>>>>>>> see exactly what was passed in. >>>>>>>>>> >>>>>>>>> Its a pretty big file. You might want to only log the >>>>>>>>> modifications. >>>>>>>>> Or save the file somewhere. >>>>>>>> >>>>>>>> Our logs are pretty verbose, so that shouldn't be a problem. I'll >>>>>>>> put it >>>>>>>> in the next version of the patch. >>>>>>> >>>>>>> The question to ask is: would you need the contents of this file if >>>>>>> all >>>>>>> you got were logs and needed to evaluate why installation failed? In >>>>>>> most cases this is yes. >>>>>>> >>>>>> Up to you guys. There is a patch I am working on in which I will be >>>>>> logging the object that is being passed to the server from pkispawn. >>>>>> That - and the diffs to the standard config file as I mentioned >>>>>> above - >>>>>> will likely be sufficient to debug almost all cases. >>>>>> >>>>>> Make sure not to log any passwords. >>>>>> >>>> >>>> Thanks for the catch. Attaching updated patch that sanitizes the >>>> passwords. >>>> >>>>>>>>>> DOGTAG: >>>>>>>>>> >>>>>>>>>> When upgrading using the dogtag-devel repo I had to specify >>>>>>>>>> pki-tools.x86_64 otherwise it tried to install both 32 and 64-bit >>>>>>>>>> versions (and failed). >>>>>>>>>> >>>>>>>>>> I ended up running: yum update pki-ca tomcatjss pki-tools.x86_64 >>>>>>>>>> --enablerepo=dogtag-devel --enablerepo=updates-testing >>>>>>>>>> >>>>>>>>> We'll look into this. I think I know why this happens. >>>>>>>>> >>>>>>>>>> What happens if someone manually upgrades pki-ca without first >>>>>>>>>> updating >>>>>>>>>> ipa? I think that pki-ca is going to need a Conflicts ipa < >>>>>>>>>> 3.0 in >>>>>>>>>> it. >>>>>>>>> >>>>>>>>> We can add that. >>>>>>>>> >>>>>>>>>> certificate renewal failed. I spent far too long trying to figure >>>>>>>>>> out >>>>>>>>>> why tomcat wasn't listening on port 9180 but failed. I think >>>>>>>>>> 9180 is >>>>>>>>>> actually the old server, right? So another missing dependency >>>>>>>>>> on a >>>>>>>>>> fixed >>>>>>>>>> certmonger? >>>>>>>>>> >>>>>>>>>> The best I could find was the certmonger error: >>>>>>>>>> >>>>>>>>>> ca-error: Error 7 connecting to >>>>>>>>>> http://edsel.example.com:9180/ca/ee/ca/profileSubmit: Couldn't >>>>>>>>>> connect >>>>>>>>>> to server. >>>>>>>>>> >>>>>>>> >>>>>>>> I'll set the certmonger min version to 0.60 as per Nalin's mail. >>>>>>> >>>>>>> Ok. >>>>>>> >>>>>>>>> Is this cert renewal on a dogtag 10 instance? Or the upgraded >>>>>>>>> dogtag 9? >>>>>>>>> If its dogtag 10, perhaps you do not have the certmonger version >>>>>>>>> that >>>>>>>>> has the relevant fix? If its dogtag 9, then we need to figure out >>>>>>>>> whats >>>>>>>>> going on. That reminds me - I need to file a bug to allow >>>>>>>>> certmonger to >>>>>>>>> talk to the newly defined dogtag ports. Do you have selinux >>>>>>>>> permissive? >>>>>>>>> >>>>>>>>>> There is no man page for pkispawn/pkidestroy :-( According to the >>>>>>>>>> FHS >>>>>>>>>> these should not be in /bin but in /usr/sbin (not end-user >>>>>>>>>> commands). >>>>>>>>>> >>>>>>>>> There is a trac ticket open for man pages for pkispawn and >>>>>>>>> pkidestroy. >>>>>>>>> We plan to complete this ticket by the time f18 is released. >>>>>>>>> >>>>>>>>> We'll look into the location of pkispawn/pkicreate. >>>>>>>>> >>>>>>>>>> The output of pkicreate/pkisilent was really terrible and not >>>>>>>>>> usable at >>>>>>>>>> all so we didn't display it when failures occurred. It looks like >>>>>>>>>> that >>>>>>>>>> has been addressed, at least for the case where a CA is already >>>>>>>>>> configured and you try to install IPA. Perhaps we should capture >>>>>>>>>> stderr >>>>>>>>>> and display that instead of the command-line of pkispawn? >>>>>>>>>> Again, a >>>>>>>>>> man >>>>>>>>>> page would help with the integration. >>>>>>>>>> >>>>>>>>>> 2012-09-10T20:51:45Z DEBUG [2/18]: configuring certificate >>>>>>>>>> server >>>>>>>>>> instance >>>>>>>>>> 2012-09-10T20:51:45Z DEBUG args=/bin/pkispawn -s CA -f >>>>>>>>>> /tmp/tmp_Urraq >>>>>>>>>> 2012-09-10T20:51:45Z DEBUG stdout= >>>>>>>>>> 2012-09-10T20:51:45Z DEBUG stderr=pkispawn : ERROR ....... >>>>>>>>>> PKI >>>>>>>>>> subsystem 'CA' for instance 'pki-tomcat' already exists! >>>>>>>>>> >>>>>>>>>> 2012-09-10T20:51:45Z CRITICAL failed to configure ca instance >>>>>>>>>> Command >>>>>>>>>> '/bin/pkispawn -s CA -f /tmp/tmp_Urraq' returned non-zero exit >>>>>>>>>> status 1 >>>>>>>>>> >>>>>>>>> That may be a good idea. Of course. thats an IPA thing, right? >>>>>>> >>>>>>> Right, not a show-stopper for this but a new ticket should be >>>>>>> opened to >>>>>>> track it. >>>> >>>> https://fedorahosted.org/freeipa/ticket/3072 >>> >>> Thanks. >>> >>> I tested a 2.2 -> 3.0 upgrade with dogtag 9 and it seems to have failed >>> to restart something: >>> >>> [ post rpm -Uvh dist/rpms/freeipa*.rpm ] >>> >>> [root at edsel freeipa]# ipa cert-show 1 >>> ipa: ERROR: Certificate operation cannot be completed: Unable to >>> communicate with CMS (Service Temporarily Unavailable) >>> [root at edsel freeipa]# ipactl restart >>> Restarting Directory Service >>> Restarting KDC Service >>> Restarting KPASSWD Service >>> Restarting MEMCACHE Service >>> Restarting HTTP Service >>> Restarting CA Service >>> [root at edsel freeipa]# ipa cert-show 1 >>> Certificate: MIIDjTCCAnWgAwIBAgIBATANBgk >>> ... >>> >>> The apache log had this: >>> >>> [Tue Sep 11 14:35:42 2012] [error] (111)Connection refused: proxy: AJP: >>> attempt to connect to 127.0.0.1:9447 (localhost) failed >>> [Tue Sep 11 14:35:42 2012] [error] ap_proxy_connect_backend disabling >>> worker for (localhost) >>> [Tue Sep 11 14:35:42 2012] [error] proxy: AJP: failed to make connection >>> to backend: localhost >>> >>> So I'm gathering that dogtag didn't restart properly, but I'm just >>> guessing. It could have been the PKI-IPA ds instance too, I'm not sure >>> where to check. >>> >>> I also noticed this in /var/log/ipaupgrade.log: >>> >>> 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl start certmonger.service >>> 2012-09-11T18:28:22Z DEBUG stdout= >>> 2012-09-11T18:28:22Z DEBUG stderr= >>> 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl is-active >>> certmonger.service >>> 2012-09-11T18:28:22Z DEBUG stdout=active >>> ... >>> ... [ snip certutil output ] >>> ... >>> 2012-09-11T18:28:52Z DEBUG args=/usr/bin/getcert start-tracking -d >>> /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c >>> dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert >>> "auditSigningCert cert-pki-ca" -P XXXXXXXX >>> 2012-09-11T18:28:52Z DEBUG stdout=Please verify that the certmonger >>> service is still running. >>> >>> 2012-09-11T18:28:52Z DEBUG stderr= >>> 2012-09-11T18:28:52Z INFO File >>> "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", >>> line 614, in run_script >>> return_value = main_function() >>> >>> File "/usr/sbin/ipa-upgradeconfig", line 540, in main >>> enable_certificate_renewal(api.env.realm) >>> >>> File "/usr/sbin/ipa-upgradeconfig", line 455, in >>> enable_certificate_renewal >>> ca.configure_renewal() >>> >>> File >>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line >>> 1298, in configure_renewal >>> self.dogtag_constants.ALIAS_DIR, 'renew_ca_cert "%s"' % nickname) >>> >>> File "/usr/lib/python2.7/site-packages/ipapython/certmonger.py", line >>> 394, in dogtag_start_tracking >>> (stdout, stderr, returncode) = ipautil.run(args, nolog=[pin]) >>> >>> File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line >>> 309, in run >>> raise CalledProcessError(p.returncode, args) >>> >>> 2012-09-11T18:28:52Z INFO The ipa-upgradeconfig command failed, >>> exception: CalledProcessError: Command '/usr/bin/getcert start-tracking >>> -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c >>> dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert >>> "auditSigningCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit >>> status 1 >>> >>> I'm not sure if this is related to this patch or not. If it isn't can >>> you file a new ticket on it, or add it the 2.2 -> 3.0 upgrade ticket? >>> >>> rob >> >> And to reply to myself, how do we imagine that upgrades will work? >> >> Is it legal for someone to upgrade to IPA 3.0 and dogtag 10 separately, >> or do we expect/require it be done at the same time, or one first? It's legal to upgrade separately, since we'll support the IPA 3.0 + Dogtag 9 combination. But to minimize the number of ways things can go wrong, we could require that IPA is upgraded before Dogtag. > Sorry to break this into a ton of e-mails. I updated pki-ca and getting > a selinux error in permissive: > > # /bin/systemctl -a status pki-cad at pki-ca.service > pki-cad at pki-ca.service - PKI Certificate Authority Server pki-ca > Loaded: loaded (/usr/lib/systemd/system/pki-cad at .service; > enabled) > Active: failed (Result: exit-code) since Tue, 11 Sep 2012 > 15:36:37 -0400; 3s ago > Process: 7067 ExecStart=/usr/bin/pkicontrol start ca %i > (code=exited, status=1/FAILURE) > Main PID: 5830 (code=exited, status=0/SUCCESS) > CGroup: name=systemd:/system/pki-cad at .service/pki-ca > > Sep 11 15:36:37 edsel.greyoak.com pkicontrol[7067]: /usr/bin/runcon: > invalid context: system_u:system_r:pki_ca_script_t:s0: Invalid argument > > I assume this means I need to re-install pki-selinux? It should be > possible to tweak the dependencies such that this package is installed > first. I'll look into it, but it would be nice to get some guidance from someone who knows more about RPMs. -- Petr? From alee at redhat.com Wed Sep 12 17:06:29 2012 From: alee at redhat.com (Ade Lee) Date: Wed, 12 Sep 2012 13:06:29 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <5050BBCB.1020500@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> <1347329065.24731.101.camel@aleeredhat.laptop> <504F318F.3030603@redhat.com> <504F35AD.3030905@redhat.com> <1347373691.24731.125.camel@aleeredhat.laptop> <504F4CE0.9080706@redhat.com> <504F67C5.4000600@redhat.com> <504F86C0.6060907@redhat.com> <504F8F8B.7040709@redhat.com> <504F9333.8040707@redhat.com> <5050BBCB.1020500@redhat.com> Message-ID: <1347469590.24731.142.camel@aleeredhat.laptop> On Wed, 2012-09-12 at 18:43 +0200, Petr Viktorin wrote: > On 09/11/2012 09:38 PM, Rob Crittenden wrote: > > Rob Crittenden wrote: > >> Rob Crittenden wrote: > >>> Petr Viktorin wrote: > >>>> On 09/11/2012 04:38 PM, Rob Crittenden wrote: > >>>>> Ade Lee wrote: > >>>>>> On Tue, 2012-09-11 at 08:59 -0400, Rob Crittenden wrote: > >>>>>>> Petr Viktorin wrote: > >>>>>>>> On 09/11/2012 04:04 AM, Ade Lee wrote: > >>>>>>>>> On Mon, 2012-09-10 at 16:58 -0400, Rob Crittenden wrote: > >>>>>>>>>> Petr Viktorin wrote: > >>>>>>>>>>> Attaching rebased and squashed patches. I've done some testing > >>>>>>>>>>> with > >>>>>>>>>>> them > >>>>>>>>>>> but please test some more. > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Most of these aren't IPA issues, but dogtag issues. I'll try to > >>>>>>>>>> split > >>>>>>>>>> them out. > >>>>>>>>>> > >>>>>>>>>> IPA: > >>>>>>>>>> > >>>>>>>>>> For the configuration files in install/conf to be updated at rpm > >>>>>>>>>> update > >>>>>>>>>> time the VERSION needs to be incremented. > >>>>>>>> > >>>>>>>> These files should stay the same since on upgrade we're still > >>>>>>>> using a > >>>>>>>> Dogtag 9 style instance. The Dogtag 10 ports are only used in new > >>>>>>>> installs. > >>>>>>>> > >>>>>>>>>> The ipa package lacks any updated dogtag dependencies, so I > >>>>>>>>>> abused > >>>>>>>>>> it. > >>>>>>>> > >>>>>>>> What should the updated dependencies be? Since it should work with > >>>>>>>> both > >>>>>>>> dogtag 9 and 10, I don't see how they should change. > >>>>>>> > >>>>>>> I don't know either, but we need to prevent people from installing > >>>>>>> incompatible package combinations. > >>>>>>> > >>>>>> Would'nt the Conflicts: ipa < 3.0 in pki-ca mentioned below satisfy > >>>>>> this > >>>>>> requirement? The main concern is that you must have ipa 3.0 if you > >>>>>> have > >>>>>> dogtag 10. > >>>>>> > >>>>>> Given that dogtag is consumed by IPA though, it makes more sense to > >>>>>> put > >>>>>> the relevant conflicts in IPA rather than in dogtag. So in this > >>>>>> case, > >>>>>> that would mean putting Conflicts: pki-ca >= 10.0 in IPA 2.x. > >>>>>> Recall that dogtag 10 will only be officially available in f18+. > >>>>> > >>>>> That isn't enough. If a F-17 user with IPA 2.2 installed upgrades to > >>>>> F-18 they would be able to install dogtag 10 and blow up their IPA > >>>>> server. > >>>>> > >>>>>> > >>>>>>>>>> I installed IPA with dogtag 9 and created a replica. > >>>>>>>>>> > >>>>>>>>>> I updated the IPA bits, that worked fine. > >>>>>>>>>> > >>>>>>>>>> I updated to dogtag 10 and now the CA doesn't work on the master, > >>>>>>>>>> including starting the dogtag instance. Note that the rpm update > >>>>>>>>>> process > >>>>>>>>>> worked, no notice that the CA service didn't restart. > >>>>>>>>>> > >>>>>>>>> Did you try to restart the CA with selinux in permissive mode? > >>>>>>>>> This is > >>>>>>>>> still required right now until I get the selinux policy all > >>>>>>>>> straightened > >>>>>>>>> out. > >>>>>>>>> > >>>>>>>>> There is also a separate dogtag ticket (which is currently being > >>>>>>>>> worked > >>>>>>>>> on) to restart existing dogtag instances when dogtag is upgraded > >>>>>>>>> from > >>>>>>>>> 9->10. > >>>>>>>> > >>>>>>>> In permissive mode, this upgrade works for me. > >>>>>>> > >>>>>>> I was in enforcing mode but saw no AVCs. What is the ETA on fixing > >>>>>>> this? > >>>>>>> > >>>>>> > >>>>>> Within the next week or two, I need to finish the IPA merge database > >>>>>> patch first, and then co-ordinate changes with the selinux guys. > >>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> Sometimes I do get this error intermittently: > >>>>>>>> > >>>>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to > >>>>>>>> communicate with CMS (Service Temporarily Unavailable) > >>>>>>>> > >>>>>>>> Usually, waiting a couple of minutes clears this up. Perhaps we are > >>>>>>>> not > >>>>>>>> doing startup detection correctly. Ade mentioned that waiting for > >>>>>>>> ports > >>>>>>>> may not be ideal. How do we know if Dogtag is initialized? > >>>>>>> > >>>>>>> Years ago we had discussed with Andrew and Matt creating a URI that > >>>>>>> can > >>>>>>> be queried to determine dogtag status. I don't think that ever went > >>>>>>> anywhere. > >>>>>>> > >>>>>> Petr, this happens only on reboot, right? And not on regular > >>>>>> "service > >>>>>> ipa restart"? > >>>> > >>>> I've now seen it happen right after a 9 ? 10 upgrade. > >>>> > >>>>>> Yeah, I remember this conversation - and even created a bug for it at > >>>>>> some point. This went away because the mechanism you were using > >>>>>> seemed > >>>>>> to be working. The timing may be very different now with tomcat 7 > >>>>>> and > >>>>>> systemd. I'll open a dogtag trac ticket for this. > >>>>> > >>>>> Ok. > >>>>> > >>>>>> > >>>>>>>> > >>>>>>>>>> Uninstalling failed because it tried to run pkidestroy and not > >>>>>>>>>> pkiremove. > >>>>>>>> > >>>>>>>> I was under the impression that pkidestroy was the correct > >>>>>>>> command to > >>>>>>>> remove an upgraded instance. I'll check with Ade. > >>>>>>>> > >>>>>>>>> I'll test this too. > >>>>>>>>> > >>>>>>>>>> The contents of the file passed to pkispawn should be logged > >>>>>>>>>> so we > >>>>>>>>>> can > >>>>>>>>>> see exactly what was passed in. > >>>>>>>>>> > >>>>>>>>> Its a pretty big file. You might want to only log the > >>>>>>>>> modifications. > >>>>>>>>> Or save the file somewhere. > >>>>>>>> > >>>>>>>> Our logs are pretty verbose, so that shouldn't be a problem. I'll > >>>>>>>> put it > >>>>>>>> in the next version of the patch. > >>>>>>> > >>>>>>> The question to ask is: would you need the contents of this file if > >>>>>>> all > >>>>>>> you got were logs and needed to evaluate why installation failed? In > >>>>>>> most cases this is yes. > >>>>>>> > >>>>>> Up to you guys. There is a patch I am working on in which I will be > >>>>>> logging the object that is being passed to the server from pkispawn. > >>>>>> That - and the diffs to the standard config file as I mentioned > >>>>>> above - > >>>>>> will likely be sufficient to debug almost all cases. > >>>>>> > >>>>>> Make sure not to log any passwords. > >>>>>> > >>>> > >>>> Thanks for the catch. Attaching updated patch that sanitizes the > >>>> passwords. > >>>> > >>>>>>>>>> DOGTAG: > >>>>>>>>>> > >>>>>>>>>> When upgrading using the dogtag-devel repo I had to specify > >>>>>>>>>> pki-tools.x86_64 otherwise it tried to install both 32 and 64-bit > >>>>>>>>>> versions (and failed). > >>>>>>>>>> > >>>>>>>>>> I ended up running: yum update pki-ca tomcatjss pki-tools.x86_64 > >>>>>>>>>> --enablerepo=dogtag-devel --enablerepo=updates-testing > >>>>>>>>>> > >>>>>>>>> We'll look into this. I think I know why this happens. > >>>>>>>>> > >>>>>>>>>> What happens if someone manually upgrades pki-ca without first > >>>>>>>>>> updating > >>>>>>>>>> ipa? I think that pki-ca is going to need a Conflicts ipa < > >>>>>>>>>> 3.0 in > >>>>>>>>>> it. > >>>>>>>>> > >>>>>>>>> We can add that. > >>>>>>>>> > >>>>>>>>>> certificate renewal failed. I spent far too long trying to figure > >>>>>>>>>> out > >>>>>>>>>> why tomcat wasn't listening on port 9180 but failed. I think > >>>>>>>>>> 9180 is > >>>>>>>>>> actually the old server, right? So another missing dependency > >>>>>>>>>> on a > >>>>>>>>>> fixed > >>>>>>>>>> certmonger? > >>>>>>>>>> > >>>>>>>>>> The best I could find was the certmonger error: > >>>>>>>>>> > >>>>>>>>>> ca-error: Error 7 connecting to > >>>>>>>>>> http://edsel.example.com:9180/ca/ee/ca/profileSubmit: Couldn't > >>>>>>>>>> connect > >>>>>>>>>> to server. > >>>>>>>>>> > >>>>>>>> > >>>>>>>> I'll set the certmonger min version to 0.60 as per Nalin's mail. > >>>>>>> > >>>>>>> Ok. > >>>>>>> > >>>>>>>>> Is this cert renewal on a dogtag 10 instance? Or the upgraded > >>>>>>>>> dogtag 9? > >>>>>>>>> If its dogtag 10, perhaps you do not have the certmonger version > >>>>>>>>> that > >>>>>>>>> has the relevant fix? If its dogtag 9, then we need to figure out > >>>>>>>>> whats > >>>>>>>>> going on. That reminds me - I need to file a bug to allow > >>>>>>>>> certmonger to > >>>>>>>>> talk to the newly defined dogtag ports. Do you have selinux > >>>>>>>>> permissive? > >>>>>>>>> > >>>>>>>>>> There is no man page for pkispawn/pkidestroy :-( According to the > >>>>>>>>>> FHS > >>>>>>>>>> these should not be in /bin but in /usr/sbin (not end-user > >>>>>>>>>> commands). > >>>>>>>>>> > >>>>>>>>> There is a trac ticket open for man pages for pkispawn and > >>>>>>>>> pkidestroy. > >>>>>>>>> We plan to complete this ticket by the time f18 is released. > >>>>>>>>> > >>>>>>>>> We'll look into the location of pkispawn/pkicreate. > >>>>>>>>> > >>>>>>>>>> The output of pkicreate/pkisilent was really terrible and not > >>>>>>>>>> usable at > >>>>>>>>>> all so we didn't display it when failures occurred. It looks like > >>>>>>>>>> that > >>>>>>>>>> has been addressed, at least for the case where a CA is already > >>>>>>>>>> configured and you try to install IPA. Perhaps we should capture > >>>>>>>>>> stderr > >>>>>>>>>> and display that instead of the command-line of pkispawn? > >>>>>>>>>> Again, a > >>>>>>>>>> man > >>>>>>>>>> page would help with the integration. > >>>>>>>>>> > >>>>>>>>>> 2012-09-10T20:51:45Z DEBUG [2/18]: configuring certificate > >>>>>>>>>> server > >>>>>>>>>> instance > >>>>>>>>>> 2012-09-10T20:51:45Z DEBUG args=/bin/pkispawn -s CA -f > >>>>>>>>>> /tmp/tmp_Urraq > >>>>>>>>>> 2012-09-10T20:51:45Z DEBUG stdout= > >>>>>>>>>> 2012-09-10T20:51:45Z DEBUG stderr=pkispawn : ERROR ....... > >>>>>>>>>> PKI > >>>>>>>>>> subsystem 'CA' for instance 'pki-tomcat' already exists! > >>>>>>>>>> > >>>>>>>>>> 2012-09-10T20:51:45Z CRITICAL failed to configure ca instance > >>>>>>>>>> Command > >>>>>>>>>> '/bin/pkispawn -s CA -f /tmp/tmp_Urraq' returned non-zero exit > >>>>>>>>>> status 1 > >>>>>>>>>> > >>>>>>>>> That may be a good idea. Of course. thats an IPA thing, right? > >>>>>>> > >>>>>>> Right, not a show-stopper for this but a new ticket should be > >>>>>>> opened to > >>>>>>> track it. > >>>> > >>>> https://fedorahosted.org/freeipa/ticket/3072 > >>> > >>> Thanks. > >>> > >>> I tested a 2.2 -> 3.0 upgrade with dogtag 9 and it seems to have failed > >>> to restart something: > >>> > >>> [ post rpm -Uvh dist/rpms/freeipa*.rpm ] > >>> > >>> [root at edsel freeipa]# ipa cert-show 1 > >>> ipa: ERROR: Certificate operation cannot be completed: Unable to > >>> communicate with CMS (Service Temporarily Unavailable) > >>> [root at edsel freeipa]# ipactl restart > >>> Restarting Directory Service > >>> Restarting KDC Service > >>> Restarting KPASSWD Service > >>> Restarting MEMCACHE Service > >>> Restarting HTTP Service > >>> Restarting CA Service > >>> [root at edsel freeipa]# ipa cert-show 1 > >>> Certificate: MIIDjTCCAnWgAwIBAgIBATANBgk > >>> ... > >>> > >>> The apache log had this: > >>> > >>> [Tue Sep 11 14:35:42 2012] [error] (111)Connection refused: proxy: AJP: > >>> attempt to connect to 127.0.0.1:9447 (localhost) failed > >>> [Tue Sep 11 14:35:42 2012] [error] ap_proxy_connect_backend disabling > >>> worker for (localhost) > >>> [Tue Sep 11 14:35:42 2012] [error] proxy: AJP: failed to make connection > >>> to backend: localhost > >>> > >>> So I'm gathering that dogtag didn't restart properly, but I'm just > >>> guessing. It could have been the PKI-IPA ds instance too, I'm not sure > >>> where to check. > >>> > >>> I also noticed this in /var/log/ipaupgrade.log: > >>> > >>> 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl start certmonger.service > >>> 2012-09-11T18:28:22Z DEBUG stdout= > >>> 2012-09-11T18:28:22Z DEBUG stderr= > >>> 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl is-active > >>> certmonger.service > >>> 2012-09-11T18:28:22Z DEBUG stdout=active > >>> ... > >>> ... [ snip certutil output ] > >>> ... > >>> 2012-09-11T18:28:52Z DEBUG args=/usr/bin/getcert start-tracking -d > >>> /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c > >>> dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert > >>> "auditSigningCert cert-pki-ca" -P XXXXXXXX > >>> 2012-09-11T18:28:52Z DEBUG stdout=Please verify that the certmonger > >>> service is still running. > >>> > >>> 2012-09-11T18:28:52Z DEBUG stderr= > >>> 2012-09-11T18:28:52Z INFO File > >>> "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", > >>> line 614, in run_script > >>> return_value = main_function() > >>> > >>> File "/usr/sbin/ipa-upgradeconfig", line 540, in main > >>> enable_certificate_renewal(api.env.realm) > >>> > >>> File "/usr/sbin/ipa-upgradeconfig", line 455, in > >>> enable_certificate_renewal > >>> ca.configure_renewal() > >>> > >>> File > >>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line > >>> 1298, in configure_renewal > >>> self.dogtag_constants.ALIAS_DIR, 'renew_ca_cert "%s"' % nickname) > >>> > >>> File "/usr/lib/python2.7/site-packages/ipapython/certmonger.py", line > >>> 394, in dogtag_start_tracking > >>> (stdout, stderr, returncode) = ipautil.run(args, nolog=[pin]) > >>> > >>> File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line > >>> 309, in run > >>> raise CalledProcessError(p.returncode, args) > >>> > >>> 2012-09-11T18:28:52Z INFO The ipa-upgradeconfig command failed, > >>> exception: CalledProcessError: Command '/usr/bin/getcert start-tracking > >>> -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c > >>> dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert > >>> "auditSigningCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit > >>> status 1 > >>> > >>> I'm not sure if this is related to this patch or not. If it isn't can > >>> you file a new ticket on it, or add it the 2.2 -> 3.0 upgrade ticket? > >>> > >>> rob > >> > >> And to reply to myself, how do we imagine that upgrades will work? > >> > >> Is it legal for someone to upgrade to IPA 3.0 and dogtag 10 separately, > >> or do we expect/require it be done at the same time, or one first? > > It's legal to upgrade separately, since we'll support the IPA 3.0 + > Dogtag 9 combination. But to minimize the number of ways things can go > wrong, we could require that IPA is upgraded before Dogtag. > > > Sorry to break this into a ton of e-mails. I updated pki-ca and getting > > a selinux error in permissive: > > > > # /bin/systemctl -a status pki-cad at pki-ca.service > > pki-cad at pki-ca.service - PKI Certificate Authority Server pki-ca > > Loaded: loaded (/usr/lib/systemd/system/pki-cad at .service; > > enabled) > > Active: failed (Result: exit-code) since Tue, 11 Sep 2012 > > 15:36:37 -0400; 3s ago > > Process: 7067 ExecStart=/usr/bin/pkicontrol start ca %i > > (code=exited, status=1/FAILURE) > > Main PID: 5830 (code=exited, status=0/SUCCESS) > > CGroup: name=systemd:/system/pki-cad at .service/pki-ca > > > > Sep 11 15:36:37 edsel.greyoak.com pkicontrol[7067]: /usr/bin/runcon: > > invalid context: system_u:system_r:pki_ca_script_t:s0: Invalid argument > > > > I assume this means I need to re-install pki-selinux? It should be > > possible to tweak the dependencies such that this package is installed > > first. > > I'll look into it, but it would be nice to get some guidance from > someone who knows more about RPMs. > > Its hard to see how this package would not be upgraded first. What seems more likely is that the semodule command loading the pki-selinux module failed to load for some reason during the upgrade. Is there any indication of what happened from the yum logs? From rcritten at redhat.com Wed Sep 12 20:25:09 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 12 Sep 2012 16:25:09 -0400 Subject: [Freeipa-devel] [PATCH] 1055 update audit cert renewal time Message-ID: <5050EFA5.7050804@redhat.com> The CA audit certificate is initially valid for two years but its profile has it renewing at six months. This bumps the value up to two years to match the other certificates. This relies on Petr's and Ade's dogtag 10 patches. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-rcrit-1055-auditrenewal.patch Type: text/x-diff Size: 4706 bytes Desc: not available URL: From jcholast at redhat.com Thu Sep 13 07:08:08 2012 From: jcholast at redhat.com (Jan Cholasta) Date: Thu, 13 Sep 2012 09:08:08 +0200 Subject: [Freeipa-devel] [PATCH] 85 Add --no-ssh option to ipa-client-install to disable OpenSSH client configuration In-Reply-To: <505093E9.3070608@redhat.com> References: <505093E9.3070608@redhat.com> Message-ID: <50518658.3030008@redhat.com> Dne 12.9.2012 15:53, Jan Cholasta napsal(a): > Hi, > > this patch fixes . > > If both --no-ssh and --no-sshd are specified, do not configure the SSH > service in SSSD. > > Honza > Note: This patch can be only applied on top of freeipa-jcholast-84-add-ssh-service-to-sssd-conf.patch Honza -- Jan Cholasta From jhrozek at redhat.com Thu Sep 13 10:28:33 2012 From: jhrozek at redhat.com (Jakub Hrozek) Date: Thu, 13 Sep 2012 12:28:33 +0200 Subject: [Freeipa-devel] [PATCH] 1054 new default SELinux user map user In-Reply-To: <504F569F.6000002@redhat.com> References: <504E5DE7.804@redhat.com> <20120910215402.GE11130@hendrix.redhat.com> <504F569F.6000002@redhat.com> Message-ID: <20120913102833.GD18572@zeppelin.brq.redhat.com> On Tue, Sep 11, 2012 at 11:19:59AM -0400, Rob Crittenden wrote: > Jakub Hrozek wrote: > >On Mon, Sep 10, 2012 at 05:38:47PM -0400, Rob Crittenden wrote: > >>We've decided to change the default SELinux user map user to the OS > >>default which is unconfined_u. It would be too drastic to go from > >>one extreme to another. > >> > >>rob > > > >How does one set an "empty default" that the SSSD would treat as "don't > >create any login file whatsoever" ? > > Patch updated to support that. > > rob > Ack, works as advertised. From mkosek at redhat.com Thu Sep 13 10:38:18 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 13 Sep 2012 12:38:18 +0200 Subject: [Freeipa-devel] [PATCH] 1054 new default SELinux user map user In-Reply-To: <20120913102833.GD18572@zeppelin.brq.redhat.com> References: <504E5DE7.804@redhat.com> <20120910215402.GE11130@hendrix.redhat.com> <504F569F.6000002@redhat.com> <20120913102833.GD18572@zeppelin.brq.redhat.com> Message-ID: <5051B79A.3050509@redhat.com> On 09/13/2012 12:28 PM, Jakub Hrozek wrote: > On Tue, Sep 11, 2012 at 11:19:59AM -0400, Rob Crittenden wrote: >> Jakub Hrozek wrote: >>> On Mon, Sep 10, 2012 at 05:38:47PM -0400, Rob Crittenden wrote: >>>> We've decided to change the default SELinux user map user to the OS >>>> default which is unconfined_u. It would be too drastic to go from >>>> one extreme to another. >>>> >>>> rob >>> >>> How does one set an "empty default" that the SSSD would treat as "don't >>> create any login file whatsoever" ? >> >> Patch updated to support that. >> >> rob >> > > Ack, > > works as advertised. > Pushed to master, ipa-3-0. Martin From mkosek at redhat.com Thu Sep 13 10:39:25 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 13 Sep 2012 12:39:25 +0200 Subject: [Freeipa-devel] [PATCH] 0078 ipa-client-install: Obtain host TGT from one specific KDC In-Reply-To: <5050B204.4010408@redhat.com> References: <5048929E.7080605@redhat.com> <504FA192.5070000@redhat.com> <50506FE3.9070505@redhat.com> <50507B6B.7050908@redhat.com> <505086EA.50402@redhat.com> <50509675.4010304@redhat.com> <5050B204.4010408@redhat.com> Message-ID: <5051B7DD.4020102@redhat.com> On 09/12/2012 06:02 PM, Petr Viktorin wrote: > On 09/12/2012 04:04 PM, Martin Kosek wrote: >> On 09/12/2012 02:58 PM, Jan Cholasta wrote: >>> Dne 12.9.2012 14:09, Petr Viktorin napsal(a): >>>> On 09/12/2012 01:20 PM, Petr Viktorin wrote: >>>>> On 09/11/2012 10:39 PM, Rob Crittenden wrote: >>>>>> Petr Viktorin wrote: >>>>>>> When installing the client, we need to take extra case to only contact >>>>>>> the one server we're installing against. Otherwise, in the real world, >>>>>>> we might hit a server that hasn't replicated info about the client yet. >>>>>>> >>>>>>> This patch fixes a bug where kinit attempted to contact a KDC that >>>>>>> didn't have the host principal yet. >>>>>>> >>>>>>> >>>>>>> To reproduce: >>>>>>> >>>>>>> - Install a "master" and "replica" >>>>>>> - Change the Kerberos DNS entries to only point to the replica: >>>>>>> for REC_NAME in '_kerberos-master._tcp' '_kerberos-master._udp' >>>>>>> '_kerberos._tcp' '_kerberos._udp' '_kpasswd._tcp' '_kpasswd._udp'; do >>>>>>> ipa dnsrecord-mod $DOMAIN $REC_NAME --srv-rec="0 100 88 >>>>>>> $REPLICA_HOSTNAME" >>>>>>> done >>>>>>> ipa dnsrecord-mod $DOMAIN _ldap._tcp --srv-rec="0 100 389 >>>>>>> $MASTER_HOSTNAME" >>>>>>> ipa dnsrecord-find $DOMAIN # check >>>>>>> - Sever communication between the hosts to disable replication: >>>>>>> (on master) >>>>>>> iptables -A INPUT -j DROP -p all --source $REPLICA_IP >>>>>>> - On client machine, put master as nameserver in /etc/resolv.conf & >>>>>>> install client >>>>>>> >>>>>>> This will fail without the patch. >>>>>>> >>>>>>> >>>>>>> Thanks to Petr Spacek, Simo, and Scott for helping to reproduce and >>>>>>> explain the bug. I learned a lot. >>>>>>> >>>>>>> https://fedorahosted.org/freeipa/ticket/2982 >>>>>> >>>>>> ACK, pushed to master and ipa-3-0 >>>>>> >>>>>> rob >>>>>> >>>>> >>>>> The patch broke server installs. Please revert it if you're having >>>>> trouble while I look into it. >>>>> >>>>> >>>> >>>> I messed up and removed the kinit call entirely when installing on >>>> master. Attaching a fix. >>>> >>> >>> Works for me, ACK. >>> >>> Honza >>> >> >> When the server installation is complete, I was surprised to see I have now >> host credentials in my CCACHE: >> >> # ipa-server-install --setup-dns >> ... >> ============================================================================== >> Setup complete >> >> Next steps: >> 1. You must make sure these network ports are open: >> TCP Ports: >> * 80, 443: HTTP/HTTPS >> * 389, 636: LDAP/LDAPS >> * 88, 464: kerberos >> * 53: bind >> UDP Ports: >> * 88, 464: kerberos >> * 53: bind >> * 123: ntp >> >> 2. You can now obtain a kerberos ticket using the command: 'kinit admin' >> This ticket will allow you to use the IPA tools (e.g., ipa user-add) >> and the web user interface. >> >> Be sure to back up the CA certificate stored in /root/cacert.p12 >> This file is required to create replicas. The password for this >> file is the Directory Manager password >> >> # klist >> Ticket cache: FILE:/tmp/krb5cc_0 >> Default principal: host/vm-086.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM >> >> Valid starting Expires Service principal >> 09/12/12 09:28:24 09/13/12 09:28:24 >> krbtgt/IDM.LAB.BOS.REDHAT.COM at IDM.LAB.BOS.REDHAT.COM >> 09/12/12 09:28:24 09/13/12 09:28:24 >> HTTP/vm-086.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM >> 09/12/12 09:28:26 09/13/12 09:28:24 >> DNS/vm-086.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM >> >> >> I don't think this is an expected behavior, installer should use a CCACHE >> separate from user's default. >> >> Martin > > I need to slow down. > Thanks for the catch. Attaching another fix. > Yup, this fixes it. ACK. Pushed to master, ipa-3-0. Martin From mkosek at redhat.com Thu Sep 13 11:15:58 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 13 Sep 2012 13:15:58 +0200 Subject: [Freeipa-devel] [PATCH] 84 Add the SSH service to SSSD config file before trying to activate it In-Reply-To: <50508990.8070700@redhat.com> References: <50508990.8070700@redhat.com> Message-ID: <5051C06E.9050902@redhat.com> On 09/12/2012 03:09 PM, Jan Cholasta wrote: > Hi, > > this patch fixes . > > Users no longer have to configure SSH in sssd.conf manually if the file exists > prior to running ipa-client-install. > > Honza > ACK. Pushed to master, ipa-3-0. Martin From mkosek at redhat.com Thu Sep 13 11:16:34 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 13 Sep 2012 13:16:34 +0200 Subject: [Freeipa-devel] [PATCH] 85 Add --no-ssh option to ipa-client-install to disable OpenSSH client configuration In-Reply-To: <505093E9.3070608@redhat.com> References: <505093E9.3070608@redhat.com> Message-ID: <5051C092.9020808@redhat.com> On 09/12/2012 03:53 PM, Jan Cholasta wrote: > Hi, > > this patch fixes . > > If both --no-ssh and --no-sshd are specified, do not configure the SSH service > in SSSD. > > Honza > ACK. Pushed to master, ipa-3-0. Martin From pspacek at redhat.com Thu Sep 13 11:36:32 2012 From: pspacek at redhat.com (Petr Spacek) Date: Thu, 13 Sep 2012 13:36:32 +0200 Subject: [Freeipa-devel] [PATCH 0060] Fix zone delete in ldap_zone_delete2() Message-ID: <5051C540.1030702@redhat.com> Hello, Fix zone delete in ldap_zone_delete2(). This fixes two race conditions during BIND reload: - failing assert in destroy_ldap_connection() DESTROYLOCK: ((pthread_mutex_destroy(&ldap_conn->lock) == 0) ? 0 : 34) == 0 - use-after-free in call: ldap_cache_enabled(cache=0xdededededededede) Petr^2 Spacek -------------- next part -------------- A non-text attachment was scrubbed... Name: bind-dyndb-ldap-pspacek-0060-Fix-zone-delete-in-ldap_zone_delete2-.-This-fixes-tw.patch Type: text/x-patch Size: 5457 bytes Desc: not available URL: From pspacek at redhat.com Thu Sep 13 11:37:37 2012 From: pspacek at redhat.com (Petr Spacek) Date: Thu, 13 Sep 2012 13:37:37 +0200 Subject: [Freeipa-devel] [PATCH 0061] Add missing DNS view attach/detach to LDAP instance management code Message-ID: <5051C581.9060209@redhat.com> Hello, Add missing DNS view attach/detach to LDAP instance management code. This fixes race condition in BIND shutdown after SIGINT: - failing assert caused by use-after-free in dns_zt_find(): (((zt) != ((void *)0)) && (((const isc__magic_t *)(zt))->magic == ((('Z') << 24 | ('T') << 16 | ('b') << 8 | ('l'))))) Petr^2 Spacek -------------- next part -------------- A non-text attachment was scrubbed... Name: bind-dyndb-ldap-pspacek-0061-Add-missing-DNS-view-attach-detach-to-LDAP-instance-.patch Type: text/x-patch Size: 1923 bytes Desc: not available URL: From mkosek at redhat.com Thu Sep 13 12:49:42 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 13 Sep 2012 14:49:42 +0200 Subject: [Freeipa-devel] [PATCH] 302 Stricter IP network validator in dnszone-add command In-Reply-To: <5047314B.30201@redhat.com> References: <5047077E.1040000@redhat.com> <50471BBD.1070506@redhat.com> <504727E5.5040101@redhat.com> <50472B1C.2070207@redhat.com> <50472DF6.8000207@redhat.com> <5047314B.30201@redhat.com> Message-ID: <5051D666.1060604@redhat.com> On 09/05/2012 01:02 PM, Jan Cholasta wrote: > Dne 5.9.2012 12:48, Martin Kosek napsal(a): >> On 09/05/2012 12:36 PM, Jan Cholasta wrote: >>> Dne 5.9.2012 12:22, Petr Spacek napsal(a): >>>> On 09/05/2012 11:30 AM, Jan Cholasta wrote: >>>>> Dne 5.9.2012 10:04, Martin Kosek napsal(a): >>>>>> We allowed IP addresses without network specification which lead >>>>>> to unexpected results when the zone was being created. We should rather >>>>>> strictly require the prefix/netmask specifying the IP network that >>>>>> the reverse zone should be created for. This is already done in >>>>>> Web UI. >>>>>> >>>>>> A unit test exercising this new validation was added. >>>>>> >>>>>> https://fedorahosted.org/freeipa/ticket/2461 >>>>>> >>>>> >>>>> I don't like this much. I would suggest using CheckedIPAddress and not >>>>> forcing >>>>> the user to enter the prefix length instead. >>>>> >>>>> CheckedIPAddress uses a sensible default prefix length if one is not >>>>> specified >>>>> (class-based for IPv4, /64 for IPv6) as opposed to IPNetwork (/32 for >>>>> IPv4, >>>>> /128 for IPv6 - this causes the erroneous reverse zones to be created as >>>>> described in the ticket). >>>>> >>>> Hello, >>>> >>>> I don't like automatic netmask guessing. I have met class-based guessing >>>> in Windows (XP?) and I was forced to overwrite default mask all the time >>>> ... >>> >>> If there was no guessing, you would have to write the netmask anyway, so I >>> don't see any harm in guessing here. >>> >>>> >>>> IMHO there is no "sensible default prefix" in real world. I sitting on >>>> network with /23 prefix right now. Also, I have never seen 10.x network >>>> with /8 prefix. >>>> >>> >>> While this might be true for IPv4 in some cases, /64 is perfectly sensible for >>> IPv6. Also, I have never seen 192.168.x.x network with non-/24 prefix. >>> >>> Honza >>> >> >> While this may be true for 192.168.x.x, it does not apply for 10.x.x.x networks >> as Petr already pointed out. I don't think that there will be many people >> expecting that a reverse zone of 10.0.0.0/24 would be created. > > And they would be correct, because the default prefix length for a class A > network is /8, not /24. > >> >> And since FreeIPA is mainly deployed to internal networks, I assume this will >> be the case of most users. >> >> Martin >> > > OK, but what about IPv6? Correct me if I'm wrong, but the prefix length is > going to be /64 99% of the time for IPv6. > > The installer uses /24 for IPv4 addresses and /64 for IPv6 addresses, maybe > this should be used as a default here as well. > > Honza > In the end, I choose a more liberal approach and instead of defining a more stricter validator for IPv4 only I rather used approach already implemented in the installers, i.e. default length of network prefix is 24 for IPv4 and 64 for IPv6. Updated patch attached. Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-302-2-use-default-reverse-zone-consistently.patch Type: text/x-patch Size: 7040 bytes Desc: not available URL: From mkosek at redhat.com Thu Sep 13 14:33:28 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 13 Sep 2012 16:33:28 +0200 Subject: [Freeipa-devel] [PATCH] 309 Fix addattr internal error Message-ID: <5051EEB8.6050303@redhat.com> When ADD command is being executed and a single-value object attribute is being set with both option and addattr IPA ends up in an internal error. Make better value sanitizing job in this case and let IPA throw a user-friendly error. Unit test exercising this situation is added. https://fedorahosted.org/freeipa/ticket/2429 -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-309-fix-addattr-internal-error.patch Type: text/x-patch Size: 2326 bytes Desc: not available URL: From mkosek at redhat.com Thu Sep 13 14:48:16 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 13 Sep 2012 16:48:16 +0200 Subject: [Freeipa-devel] [PATCH] 213 JSON serialization of long type In-Reply-To: <504F54B5.6040901@redhat.com> References: <504F54B5.6040901@redhat.com> Message-ID: <5051F230.6090004@redhat.com> On 09/11/2012 05:11 PM, Petr Vobornik wrote: > Numbers of long type were incorrectly serialized to JSON as empty strings when > using json_serialize function. It caused problem in serialization of metadata > for Web UI. This patch is fixing it. > > Discovered after "Cast DNS SOA serial maximum boundary to long" > ACK. Pushed to master, ipa-3-0. Martin From rcritten at redhat.com Thu Sep 13 16:40:16 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 13 Sep 2012 12:40:16 -0400 Subject: [Freeipa-devel] [PATCH] 305-308 Expand Referential Integrity checks In-Reply-To: <50507B31.6070804@redhat.com> References: <50507B31.6070804@redhat.com> Message-ID: <50520C70.7080800@redhat.com> Martin Kosek wrote: > To test, add sudo commands, hosts or users to a sudo rule or hbac rule and then > rename or delete the linked object. After the update, the links should be amended. > > --------- > > Many attributes in IPA (e.g. manager, memberuser, managedby, ...) > are used to store DNs of linked objects in IPA (users, hosts, sudo > commands, etc.). However, when the linked objects is deleted or > renamed, the attribute pointing to it stays with the objects and > thus may create a dangling link causing issues in client software > reading the data. > > Directory Server has a plugin to enforce referential integrity (RI) > by checking DEL and MODRDN operations and updating affected links. > It was already used for manager and secretary attributes and > should be expanded for the missing attributes to avoid dangling > links. > > As a prerequisite, all attributes checked for RI must have pres > and eq indexes to avoid performance issues. The following indexes > have been added: > * manager (pres index only) > * secretary (pres index only) > * memberHost > * memberUser > * sourcehost > * memberservice > * managedby > * memberallowcmd > * memberdenycmd > * ipasudorunas > * ipasudorunasgroup > > Referential Integrity plugin was updated to check all these > attributes. > > Note: this update will only fix RI on one master as RI plugin does > not check replicated operations. > > https://fedorahosted.org/freeipa/ticket/2866 These patches look good but I'd like to see some tests associated with the referential integrity changes in patch 308. I'm not sure we need a test for every single combination where RI comes into play but at least testing that the original sequence (sudorule/sudocmd) works as expected. rob From rcritten at redhat.com Thu Sep 13 19:14:27 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 13 Sep 2012 15:14:27 -0400 Subject: [Freeipa-devel] [PATCH] 1056 sudorule cn uniqueness Message-ID: <50523093.7040704@redhat.com> A sudorule dn uses ipaUniqueId as the cn so we have to do a search to ensure uniqueness. This leaves us vulnerable to a race. Configure the uniqueness plugin to ensure no dups. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-rcrit-1056-sudorule-uniqueness.patch Type: text/x-diff Size: 3226 bytes Desc: not available URL: From rcritten at redhat.com Thu Sep 13 19:19:34 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 13 Sep 2012 15:19:34 -0400 Subject: [Freeipa-devel] [PATCH] 309 Fix addattr internal error In-Reply-To: <5051EEB8.6050303@redhat.com> References: <5051EEB8.6050303@redhat.com> Message-ID: <505231C6.9060105@redhat.com> Martin Kosek wrote: > When ADD command is being executed and a single-value object attribute > is being set with both option and addattr IPA ends up in an internal > error. > > Make better value sanitizing job in this case and let IPA throw > a user-friendly error. Unit test exercising this situation is added. > > https://fedorahosted.org/freeipa/ticket/2429 + if not isinstance(val, (list, tuple)): + val = [val] + val.extend(adddict[attr]) I val is a tuple the extend is still going to fail. Can val ever be a tuple? If so we'd need to convert it to a list. rob From rcritten at redhat.com Thu Sep 13 19:21:19 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 13 Sep 2012 15:21:19 -0400 Subject: [Freeipa-devel] [PATCH] 0079 Update the pot file (translation source) In-Reply-To: <50504AEF.9090209@redhat.com> References: <50504AEF.9090209@redhat.com> Message-ID: <5052322F.2040009@redhat.com> Petr Viktorin wrote: > Transifex is watching our repository, so pushing this patch will update > the translations on the site. ACK, pushed to master and ipa-3-0 rob From rcritten at redhat.com Thu Sep 13 20:35:19 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 13 Sep 2012 16:35:19 -0400 Subject: [Freeipa-devel] [PATCH] 0077 Check direct/reverse hostname/address resolution in ipa-replica-install In-Reply-To: <5050B1ED.70307@redhat.com> References: <5046009E.2020306@redhat.com> <50463DE6.9070300@redhat.com> <5046FF0E.1000504@redhat.com> <504FA7A3.5080201@redhat.com> <5050B1ED.70307@redhat.com> Message-ID: <50524387.4020800@redhat.com> Petr Viktorin wrote: > On 09/11/2012 11:05 PM, Rob Crittenden wrote: >> Petr Viktorin wrote: >>> On 09/04/2012 07:44 PM, Rob Crittenden wrote: >>>> Petr Viktorin wrote: >>>>> >>>>> https://fedorahosted.org/freeipa/ticket/2845 >>>> >>>> Shouldn't this also call verify_fqdn() on the local hostname and not >>>> just the master? I think this would eventually fail in the conncheck >>>> but >>>> what if that was skipped? >>>> >>>> rob >>> >>> A few lines above there is a call to get_host_name, which will call >>> verify_fqdn. >>> >> >> I double-checked this, it fails in conncheck. Here are my steps: >> >> # ipa-server-install --setup-dns >> # ipa-replica-prepare replica.example.com --ip-address=192.168.100.2 >> # ipa host-del replica.example.com >> >> On replica, set DNS to IPA master, with hostname in /etc/hosts. >> >> # ipa-replica-install ... >> >> The verify_fqdn() passes because the resolver uses /etc/hosts. >> >> The conncheck fails: >> >> Execute check on remote master >> Check connection from master to remote replica 'replica.example.com': >> >> Remote master check failed with following error message(s): >> Could not chdir to home directory /home/admin: No such file or directory >> Port check failed! Unable to resolve host name 'replica.example.com' >> >> Connection check failed! >> Please fix your network settings according to error messages above. >> If the check results are not valid it can be skipped with >> --skip-conncheck parameter. >> >> The DNS test happens much further after this, and I get why, I just >> don't see how useful it is unless the --skip-conncheck is used. > > For the record, it's because we need to check if the host has DNS > installed. We need a LDAP connection to check this. > >> ipa-replica-install ~rcrit/replica-info-replica.example.com.gpg >> --skip-conncheck >> Directory Manager (existing master) password: >> >> ipa : ERROR Could not resolve hostname replica.example.com >> using DNS. Clients may not function properly. Please check your DNS >> setup. (Note that this check queries IPA DNS directly and ignores >> /etc/hosts.) >> Continue? [no]: >> >> So I guess, what are the intentions here? It is certainly better than >> before. >> >> rob > > If the replica is in the master's /etc/hosts, but not in DNS, the > conncheck will succeed. This check explicitly queries IPA records only > and ignores /etc/hosts so it'll notice this case and warn. > Ok, like I said, this is better than we have. Just one nit then you get an ack: + # If remote host has DNS, check forward/reverse resolution + try: + entry = conn.find_entries(u'cn=dns', base_dn=DN(api.env.basedn)) + except errors.NotFound: u'cn=dns' should be str(constants.container_dns). rob From mkosek at redhat.com Fri Sep 14 06:40:32 2012 From: mkosek at redhat.com (Martin Kosek) Date: Fri, 14 Sep 2012 08:40:32 +0200 Subject: [Freeipa-devel] [PATCH] 309 Fix addattr internal error In-Reply-To: <505231C6.9060105@redhat.com> References: <5051EEB8.6050303@redhat.com> <505231C6.9060105@redhat.com> Message-ID: <5052D160.1000701@redhat.com> On 09/13/2012 09:19 PM, Rob Crittenden wrote: > Martin Kosek wrote: >> When ADD command is being executed and a single-value object attribute >> is being set with both option and addattr IPA ends up in an internal >> error. >> >> Make better value sanitizing job in this case and let IPA throw >> a user-friendly error. Unit test exercising this situation is added. >> >> https://fedorahosted.org/freeipa/ticket/2429 > > + if not isinstance(val, (list, tuple)): > + val = [val] > + val.extend(adddict[attr]) > > I val is a tuple the extend is still going to fail. Can val ever be a tuple? If > so we'd need to convert it to a list. > > rob I don't think it could be a tuple, I am about 99% certain. So for this 1% I added a special clause for tuple. Patch attached. We will be able to be even more certain when Honza finishes his strict encoding patch he was working on in the summer. With his patch, the attributes should always be a list. Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-309-2-fix-addattr-internal-error.patch Type: text/x-patch Size: 2411 bytes Desc: not available URL: From mkosek at redhat.com Fri Sep 14 06:46:36 2012 From: mkosek at redhat.com (Martin Kosek) Date: Fri, 14 Sep 2012 08:46:36 +0200 Subject: [Freeipa-devel] [PATCH] 0077 Check direct/reverse hostname/address resolution in ipa-replica-install In-Reply-To: <50524387.4020800@redhat.com> References: <5046009E.2020306@redhat.com> <50463DE6.9070300@redhat.com> <5046FF0E.1000504@redhat.com> <504FA7A3.5080201@redhat.com> <5050B1ED.70307@redhat.com> <50524387.4020800@redhat.com> Message-ID: <5052D2CC.2000405@redhat.com> On 09/13/2012 10:35 PM, Rob Crittenden wrote: > Petr Viktorin wrote: >> On 09/11/2012 11:05 PM, Rob Crittenden wrote: >>> Petr Viktorin wrote: >>>> On 09/04/2012 07:44 PM, Rob Crittenden wrote: >>>>> Petr Viktorin wrote: >>>>>> >>>>>> https://fedorahosted.org/freeipa/ticket/2845 >>>>> >>>>> Shouldn't this also call verify_fqdn() on the local hostname and not >>>>> just the master? I think this would eventually fail in the conncheck >>>>> but >>>>> what if that was skipped? >>>>> >>>>> rob >>>> >>>> A few lines above there is a call to get_host_name, which will call >>>> verify_fqdn. >>>> >>> >>> I double-checked this, it fails in conncheck. Here are my steps: >>> >>> # ipa-server-install --setup-dns >>> # ipa-replica-prepare replica.example.com --ip-address=192.168.100.2 >>> # ipa host-del replica.example.com >>> >>> On replica, set DNS to IPA master, with hostname in /etc/hosts. >>> >>> # ipa-replica-install ... >>> >>> The verify_fqdn() passes because the resolver uses /etc/hosts. >>> >>> The conncheck fails: >>> >>> Execute check on remote master >>> Check connection from master to remote replica 'replica.example.com': >>> >>> Remote master check failed with following error message(s): >>> Could not chdir to home directory /home/admin: No such file or directory >>> Port check failed! Unable to resolve host name 'replica.example.com' >>> >>> Connection check failed! >>> Please fix your network settings according to error messages above. >>> If the check results are not valid it can be skipped with >>> --skip-conncheck parameter. >>> >>> The DNS test happens much further after this, and I get why, I just >>> don't see how useful it is unless the --skip-conncheck is used. >> >> For the record, it's because we need to check if the host has DNS >> installed. We need a LDAP connection to check this. >> >>> ipa-replica-install ~rcrit/replica-info-replica.example.com.gpg >>> --skip-conncheck >>> Directory Manager (existing master) password: >>> >>> ipa : ERROR Could not resolve hostname replica.example.com >>> using DNS. Clients may not function properly. Please check your DNS >>> setup. (Note that this check queries IPA DNS directly and ignores >>> /etc/hosts.) >>> Continue? [no]: >>> >>> So I guess, what are the intentions here? It is certainly better than >>> before. >>> >>> rob >> >> If the replica is in the master's /etc/hosts, but not in DNS, the >> conncheck will succeed. This check explicitly queries IPA records only >> and ignores /etc/hosts so it'll notice this case and warn. >> > > Ok, like I said, this is better than we have. Just one nit then you get an ack: > > + # If remote host has DNS, check forward/reverse resolution > + try: > + entry = conn.find_entries(u'cn=dns', base_dn=DN(api.env.basedn)) > + except errors.NotFound: > > u'cn=dns' should be str(constants.container_dns). > > rob This is a search filter, Petr could use the one I already have in "dns.py::get_dns_masters()" function: '(&(objectClass=ipaConfigObject)(cn=DNS))' For performance sake, I would also not search in the entire tree, but limit the search only to: DN(('cn', 'masters'), ('cn', 'ipa'), ('cn', 'etc'), api.env.basedn) Martin From mkosek at redhat.com Fri Sep 14 08:40:59 2012 From: mkosek at redhat.com (Martin Kosek) Date: Fri, 14 Sep 2012 10:40:59 +0200 Subject: [Freeipa-devel] [PATCH] 305-308 Expand Referential Integrity checks In-Reply-To: <50520C70.7080800@redhat.com> References: <50507B31.6070804@redhat.com> <50520C70.7080800@redhat.com> Message-ID: <5052ED9B.7000609@redhat.com> On 09/13/2012 06:40 PM, Rob Crittenden wrote: > Martin Kosek wrote: >> To test, add sudo commands, hosts or users to a sudo rule or hbac rule and then >> rename or delete the linked object. After the update, the links should be >> amended. >> >> --------- >> >> Many attributes in IPA (e.g. manager, memberuser, managedby, ...) >> are used to store DNs of linked objects in IPA (users, hosts, sudo >> commands, etc.). However, when the linked objects is deleted or >> renamed, the attribute pointing to it stays with the objects and >> thus may create a dangling link causing issues in client software >> reading the data. >> >> Directory Server has a plugin to enforce referential integrity (RI) >> by checking DEL and MODRDN operations and updating affected links. >> It was already used for manager and secretary attributes and >> should be expanded for the missing attributes to avoid dangling >> links. >> >> As a prerequisite, all attributes checked for RI must have pres >> and eq indexes to avoid performance issues. The following indexes >> have been added: >> * manager (pres index only) >> * secretary (pres index only) >> * memberHost >> * memberUser >> * sourcehost >> * memberservice >> * managedby >> * memberallowcmd >> * memberdenycmd >> * ipasudorunas >> * ipasudorunasgroup >> >> Referential Integrity plugin was updated to check all these >> attributes. >> >> Note: this update will only fix RI on one master as RI plugin does >> not check replicated operations. >> >> https://fedorahosted.org/freeipa/ticket/2866 > > These patches look good but I'd like to see some tests associated with the > referential integrity changes in patch 308. I'm not sure we need a test for > every single combination where RI comes into play but at least testing that the > original sequence (sudorule/sudocmd) works as expected. > > rob Right, I should have seen that coming. I want this feature to be checked properly so I added a tests for all RI-checked attributes. Patches attached. Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-305-2-add-attributeTypes-to-safe-schema-updater.patch Type: text/x-patch Size: 6467 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-306-2-amend-memberallowcmd-and-memberdenycmd-attribute-typ.patch Type: text/x-patch Size: 4864 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-307-2-run-index-task-in-ldap-updater-only-when-needed.patch Type: text/x-patch Size: 2813 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-308-2-expand-referential-integrity-checks.patch Type: text/x-patch Size: 25021 bytes Desc: not available URL: From pspacek at redhat.com Fri Sep 14 08:50:56 2012 From: pspacek at redhat.com (Petr Spacek) Date: Fri, 14 Sep 2012 10:50:56 +0200 Subject: [Freeipa-devel] [PATCH 0062] Prevent memory read outside allocated space in str_alloc() Message-ID: <5052EFF0.1040805@redhat.com> Hello, Prevent memory read outside allocated space in str_alloc(). Found by Valgrind during nsupdate stress test. -- Petr^2 Spacek -------------- next part -------------- A non-text attachment was scrubbed... Name: bind-dyndb-ldap-pspacek-0062-Prevent-memory-read-outside-allocated-space-in-str_a.patch Type: text/x-patch Size: 817 bytes Desc: not available URL: From pviktori at redhat.com Fri Sep 14 11:51:36 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Fri, 14 Sep 2012 13:51:36 +0200 Subject: [Freeipa-devel] [PATCH] 0079 Update the pot file (translation source) In-Reply-To: <5052322F.2040009@redhat.com> References: <50504AEF.9090209@redhat.com> <5052322F.2040009@redhat.com> Message-ID: <50531A48.9080604@redhat.com> On 09/13/2012 09:21 PM, Rob Crittenden wrote: > Petr Viktorin wrote: >> Transifex is watching our repository, so pushing this patch will update >> the translations on the site. Okay, I lied. Some time ago (before I joined), Transifex changed its "watching" mechanism from VCS pulls to URL polling. I guess IPA got "unwatched" then. I pushed the pot manually. Since we have infrequent explicit string freezes I don't think it's necessary to configure automatic pot updates again. > > ACK, pushed to master and ipa-3-0 > > rob -- Petr? From mkosek at redhat.com Fri Sep 14 11:56:44 2012 From: mkosek at redhat.com (Martin Kosek) Date: Fri, 14 Sep 2012 13:56:44 +0200 Subject: [Freeipa-devel] [PATCH] 1050 prevent replica orphans In-Reply-To: <504E329D.6070905@redhat.com> References: <5040F6E2.7060109@redhat.com> <5040F722.4060208@redhat.com> <5048BFAD.1070807@redhat.com> <50491414.2070504@redhat.com> <1347008704.2829.13.camel@priserak> <504E329D.6070905@redhat.com> Message-ID: <50531B7C.8070401@redhat.com> On 09/10/2012 08:34 PM, Rob Crittenden wrote: > Martin Kosek wrote: >> On Thu, 2012-09-06 at 17:22 -0400, Rob Crittenden wrote: >>> Martin Kosek wrote: >>>> On 08/31/2012 07:40 PM, Rob Crittenden wrote: >>>>> Rob Crittenden wrote: >>>>>> It was possible use ipa-replica-manage connect/disconnect/del to end up >>>>>> orphaning or or more IPA masters. This is an attempt to catch and >>>>>> prevent that case. >>>>>> >>>>>> I tested with this topology, trying to delete B. >>>>>> >>>>>> A <-> B <-> C >>>>>> >>>>>> I got here by creating B and C from A, connecting B to C then deleting >>>>>> the link from A to B, so it went from A -> B and A -> C to the above. >>>>>> >>>>>> What I do is look up the servers that the delete candidate host has >>>>>> connections to and see if we're the last link. >>>>>> >>>>>> I added an escape clause if there are only two masters. >>>>>> >>>>>> rob >>>>> >>>>> Oh, this relies on my cleanruv patch 1031. >>>>> >>>>> rob >>>>> >>>> >>>> 1) When I run ipa-replica-manage del --force to an already uninstalled host, >>>> the new code will prevent me the deletation because it cannot connect to >>>> it. It >>>> also crashes with UnboundLocalError: >>>> >>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>> >>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing removal >>>> Traceback (most recent call last): >>>> File "/sbin/ipa-replica-manage", line 708, in >>>> main() >>>> File "/sbin/ipa-replica-manage", line 677, in main >>>> del_master(realm, args[1], options) >>>> File "/sbin/ipa-replica-manage", line 476, in del_master >>>> sys.exit("Failed read master data from '%s': %s" % (delrepl.hostname, >>>> str(e))) >>>> UnboundLocalError: local variable 'delrepl' referenced before assignment >>> >>> Fixed. >>> >>>> >>>> >>>> I also hit this error when removing a winsync replica. >>> >>> Fixed. >>> >>>> >>>> >>>> 2) As I wrote before, I think having --force option override the user >>>> inquiries >>>> would benefit test automation: >>>> >>>> + if not ipautil.user_input("Continue to delete?", False): >>>> + sys.exit("Aborted") >>> >>> Fixed. >>> >>>> >>>> >>>> 3) I don't think this code won't cover this topology: >>>> >>>> A - B - C - D - E >>>> >>>> It would allow you deleting a replica C even though it would separate A-B and >>>> D-E. Though we may not want to cover this situation now, what you got is >>>> definitely helping. >>> >>> I think you may be right. I only tested with 4 servers. With this B and >>> D would both still have 2 agreements so wouldn't be covered by the last >>> link test. >> >> Everything looks good now, so ACK. We just need to push it along with >> CLEANALLRUV patch. >> >> Martin >> > > Not to look a gift ACK In the mouth but here is a revised patch. I've added a > cleanup routine to remove an orphaned master properly. If you had tried the > mechanism I outlined in the man page it would have worked but was > less-than-complete. This way is better, just don't try it on a live master. > > I also added a cleanruv abort command, in case you want to kill an existing task. > > rob > 1) CLEANRUV stuff should be in your patch 1031 and not here (but I will comment on the code in this mail since it is in this patch) 2) In new command defitinion: + "abort-clean-ruv":(1, 1, "Replica ID of to abort cleaning", ""), I miss error message in case REPLICA_ID is not passed, this way error message when I omit the parameter is confusing: # ipa-replica-manage abort-clean-ruv Usage: ipa-replica-manage [options] ipa-replica-manage: error: must provide a command [force-sync | clean-ruv | disconnect | connect | list-ruv | del | re-initialize | list | abort-clean-ruv] On another note, I am thinking about the new command(s). Since we now have abort-clean-ruv command, we may want to also implement list-clean-ruv commands in the future to see all CLEANALLRUV commands in process... Or we may enhance list-ruv to write a flag like [CLEANALLRUV in process] for RUV's for which CLEANALLRUV task is in process. 3) Will clean-ruv - abort-clean-ruv - clean-ruv sequence work? If the aborted CLEANALLRUV task stays in DIT, we may not be able to enter new CLEANALLRUV task because we always use the same DN... Btw. did abort CLEANALLRUV command worked for you? Mine seemed to be stuck on replicas that are down just like CLEANALLRUV command. I had then paralelly running CLEANALLRUV and ABORT CLEANALLRUV running for the same RUV ID. Then, it is unclear to me what this command is actually good for. 4) Man page now contains non-existent command: --- a/install/tools/man/ipa-replica-manage.1 +++ b/install/tools/man/ipa-replica-manage.1 @@ -42,12 +42,18 @@ Manages the replication agreements of an IPA server. \fBforce\-sync\fR \- Immediately flush any data to be replicated from a server specified with the \-\-from option .TP +\fBcleanup\fR +\- Remove leftover references to a deleted master. +.TP The cleanup procedure was implemented rather as an option to the del command than a separate command. 5) My favorite - new cleanup procedure user prompt miss the --force option useful for test automation + if not ipautil.user_input("Continue to clean master?", False): + sys.exit("Cleanup aborted") Otherwise the patch looks good. Martin From mkosek at redhat.com Fri Sep 14 12:06:52 2012 From: mkosek at redhat.com (Martin Kosek) Date: Fri, 14 Sep 2012 14:06:52 +0200 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <504912F9.1020907@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> <50479491.1060200@redhat.com> <50488DD0.1080507@redhat.com> <5048BEF8.5000401@redhat.com> <5048C46A.8020207@redhat.com> <5048C754.2030708@redhat.com> <5048C9BE.1050005@redhat.com> <504912F9.1020907@redhat.com> Message-ID: <50531DDC.5050007@redhat.com> On 09/06/2012 11:17 PM, Rob Crittenden wrote: > Martin Kosek wrote: >> On 09/06/2012 05:55 PM, Rob Crittenden wrote: >>> Rob Crittenden wrote: >>>> Rob Crittenden wrote: >>>>> Martin Kosek wrote: >>>>>> On 09/05/2012 08:06 PM, Rob Crittenden wrote: >>>>>>> Rob Crittenden wrote: >>>>>>>> Martin Kosek wrote: >>>>>>>>> On 07/05/2012 08:39 PM, Rob Crittenden wrote: >>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: >>>>>>>>>>>> Deleting a replica can leave a replication vector (RUV) on the >>>>>>>>>>>> other servers. >>>>>>>>>>>> This can confuse things if the replica is re-added, and it also >>>>>>>>>>>> causes the >>>>>>>>>>>> server to calculate changes against a server that may no longer >>>>>>>>>>>> exist. >>>>>>>>>>>> >>>>>>>>>>>> 389-ds-base provides a new task that self-propogates itself to all >>>>>>>>>>>> available >>>>>>>>>>>> replicas to clean this RUV data. >>>>>>>>>>>> >>>>>>>>>>>> This patch will create this task at deletion time to hopefully >>>>>>>>>>>> clean things up. >>>>>>>>>>>> >>>>>>>>>>>> It isn't perfect. If any replica is down or unavailable at the >>>>>>>>>>>> time >>>>>>>>>>>> the >>>>>>>>>>>> cleanruv task fires, and then comes back up, the old RUV data >>>>>>>>>>>> may be >>>>>>>>>>>> re-propogated around. >>>>>>>>>>>> >>>>>>>>>>>> To make things easier in this case I've added two new commands to >>>>>>>>>>>> ipa-replica-manage. The first lists the replication ids of all the >>>>>>>>>>>> servers we >>>>>>>>>>>> have a RUV for. Using this you can call clean_ruv with the >>>>>>>>>>>> replication id of a >>>>>>>>>>>> server that no longer exists to try the cleanallruv step again. >>>>>>>>>>>> >>>>>>>>>>>> This is quite dangerous though. If you run cleanruv against a >>>>>>>>>>>> replica id that >>>>>>>>>>>> does exist it can cause a loss of data. I believe I've put in >>>>>>>>>>>> enough scary >>>>>>>>>>>> warnings about this. >>>>>>>>>>>> >>>>>>>>>>>> rob >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Good work there, this should make cleaning RUVs much easier than >>>>>>>>>>> with the >>>>>>>>>>> previous version. >>>>>>>>>>> >>>>>>>>>>> This is what I found during review: >>>>>>>>>>> >>>>>>>>>>> 1) list_ruv and clean_ruv command help in man is quite lost. I >>>>>>>>>>> think >>>>>>>>>>> it would >>>>>>>>>>> help if we for example have all info for commands indented. This >>>>>>>>>>> way >>>>>>>>>>> user could >>>>>>>>>>> simply over-look the new commands in the man page. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2) I would rename new commands to clean-ruv and list-ruv to make >>>>>>>>>>> them >>>>>>>>>>> consistent with the rest of the commands (re-initialize, >>>>>>>>>>> force-sync). >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 3) It would be nice to be able to run clean_ruv command in an >>>>>>>>>>> unattended way >>>>>>>>>>> (for better testing), i.e. respect --force option as we already >>>>>>>>>>> do for >>>>>>>>>>> ipa-replica-manage del. This fix would aid test automation in the >>>>>>>>>>> future. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 4) (minor) The new question (and the del too) does not react too >>>>>>>>>>> well for >>>>>>>>>>> CTRL+D: >>>>>>>>>>> >>>>>>>>>>> # ipa-replica-manage clean_ruv 3 --force >>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>> >>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>> Continue to clean? [no]: unexpected error: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 5) Help for clean_ruv command without a required parameter is quite >>>>>>>>>>> confusing >>>>>>>>>>> as it reports that command is wrong and not the parameter: >>>>>>>>>>> >>>>>>>>>>> # ipa-replica-manage clean_ruv >>>>>>>>>>> Usage: ipa-replica-manage [options] >>>>>>>>>>> >>>>>>>>>>> ipa-replica-manage: error: must provide a command [clean_ruv | >>>>>>>>>>> force-sync | >>>>>>>>>>> disconnect | connect | del | re-initialize | list | list_ruv] >>>>>>>>>>> >>>>>>>>>>> It seems you just forgot to specify the error message in the >>>>>>>>>>> command >>>>>>>>>>> definition >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 6) When the remote replica is down, the clean_ruv command fails >>>>>>>>>>> with an >>>>>>>>>>> unexpected error: >>>>>>>>>>> >>>>>>>>>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 >>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>> >>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>> Continue to clean? [no]: y >>>>>>>>>>> unexpected error: {'desc': 'Operations error'} >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: >>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>> cleanAllRUV_task: failed >>>>>>>>>>> to connect to repl agreement connection >>>>>>>>>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, >>>>>>>>>>> >>>>>>>>>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> tree,cn=config), error 105 >>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>> cleanAllRUV_task: replica >>>>>>>>>>> (cn=meTovm-055.idm.lab. >>>>>>>>>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> tree, cn=config) has not been cleaned. You will need to rerun >>>>>>>>>>> the >>>>>>>>>>> CLEANALLRUV task on this replica. >>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>> cleanAllRUV_task: Task >>>>>>>>>>> failed (1) >>>>>>>>>>> >>>>>>>>>>> In this case I think we should inform user that the command failed, >>>>>>>>>>> possibly >>>>>>>>>>> because of disconnected replicas and that they could enable the >>>>>>>>>>> replicas and >>>>>>>>>>> try again. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 7) (minor) "pass" is now redundant in replication.py: >>>>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>>>>> + # We can't make the server we're removing read-only >>>>>>>>>>> but >>>>>>>>>>> + # this isn't a show-stopper >>>>>>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>>>>>> read-only, >>>>>>>>>>> continuing anyway") >>>>>>>>>>> + pass >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I think this addresses everything. >>>>>>>>>> >>>>>>>>>> rob >>>>>>>>> >>>>>>>>> Thanks, almost there! I just found one more issue which needs to be >>>>>>>>> fixed >>>>>>>>> before we push: >>>>>>>>> >>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>>>>>>> Directory Manager password: >>>>>>>>> >>>>>>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing >>>>>>>>> removal >>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>>> "Can't >>>>>>>>> contact LDAP server"} >>>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>>> >>>>>>>>> There were issues removing a connection: %d format: a number is >>>>>>>>> required, not str >>>>>>>>> >>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>>> "Can't >>>>>>>>> contact LDAP server"} >>>>>>>>> >>>>>>>>> This is a traceback I retrieved: >>>>>>>>> Traceback (most recent call last): >>>>>>>>> File "/sbin/ipa-replica-manage", line 425, in del_master >>>>>>>>> del_link(realm, r, hostname, options.dirman_passwd, force=True) >>>>>>>>> File "/sbin/ipa-replica-manage", line 271, in del_link >>>>>>>>> repl1.cleanallruv(replica_id) >>>>>>>>> File >>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>>>>>> line 1094, in cleanallruv >>>>>>>>> root_logger.debug("Creating CLEANALLRUV task for replica id >>>>>>>>> %d" % >>>>>>>>> replicaId) >>>>>>>>> >>>>>>>>> >>>>>>>>> The problem here is that you don't convert replica_id to int in this >>>>>>>>> part: >>>>>>>>> + replica_id = None >>>>>>>>> + if repl2: >>>>>>>>> + replica_id = repl2._get_replica_id(repl2.conn, None) >>>>>>>>> + else: >>>>>>>>> + servers = get_ruv(realm, replica1, dirman_passwd) >>>>>>>>> + for (netloc, rid) in servers: >>>>>>>>> + if netloc.startswith(replica2): >>>>>>>>> + replica_id = rid >>>>>>>>> + break >>>>>>>>> >>>>>>>>> Martin >>>>>>>>> >>>>>>>> >>>>>>>> Updated patch using new mechanism in 389-ds-base. This should more >>>>>>>> thoroughly clean out RUV data when a replica is being deleted, and >>>>>>>> provide for a way to delete RUV data afterwards too if necessary. >>>>>>>> >>>>>>>> rob >>>>>>> >>>>>>> Rebased patch >>>>>>> >>>>>>> rob >>>>>>> >>>>>> >>>>>> 0) As I wrote in a review for your patch 1041, changelog entry slipped >>>>>> elsewhere. >>>>>> >>>>>> 1) The following KeyboardInterrupt except class looks suspicious. I >>>>>> know why >>>>>> you have it there, but since it is generally a bad thing to do, some >>>>>> comment >>>>>> why it is needed would be useful. >>>>>> >>>>>> @@ -256,6 +263,17 @@ def del_link(realm, replica1, replica2, >>>>>> dirman_passwd, >>>>>> force=False): >>>>>> repl1.delete_agreement(replica2) >>>>>> repl1.delete_referral(replica2) >>>>>> >>>>>> + if type1 == replication.IPA_REPLICA: >>>>>> + if repl2: >>>>>> + ruv = repl2._get_replica_id(repl2.conn, None) >>>>>> + else: >>>>>> + ruv = get_ruv_by_host(realm, replica1, replica2, >>>>>> dirman_passwd) >>>>>> + >>>>>> + try: >>>>>> + repl1.cleanallruv(ruv) >>>>>> + except KeyboardInterrupt: >>>>>> + pass >>>>>> + >>>>>> >>>>>> Maybe you just wanted to do some cleanup and then "raise" again? >>>>> >>>>> No, it is there because it is safe to break out of it. The task will >>>>> continue to run. I added some verbiage. >>>>> >>>>>> >>>>>> 2) This is related to 1), but when some replica is down, >>>>>> "ipa-replica-manage >>>>>> del" may wait indefinitely when some remote replica is down, right? >>>>>> >>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>> Deleting a master is irreversible. >>>>>> To reconnect to the remote master you will need to prepare a new >>>>>> replica file >>>>>> and re-install. >>>>>> Continue to delete? [no]: y >>>>>> ipa: INFO: Setting agreement >>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> tree,cn=config >>>>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica >>>>>> acquired >>>>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>>>> Background task created to clean replication data >>>>>> >>>>>> ... after about a minute I hit CTRL+C >>>>>> >>>>>> ^CDeleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to >>>>>> 'vm-055.idm.lab.bos.redhat.com' >>>>>> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: NS record >>>>>> does not >>>>>> contain 'vm-055.idm.lab.bos.redhat.com.' >>>>>> You may need to manually remove them from the tree >>>>>> >>>>>> I think it would be better to inform user that some remote replica is >>>>>> down or >>>>>> at least that we are waiting for the task to complete. Something like >>>>>> that: >>>>>> >>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>> ... >>>>>> Background task created to clean replication data >>>>>> Replication data clean up may take very long time if some replica is >>>>>> unreachable >>>>>> Hit CTRL+C to interrupt the wait >>>>>> ^C Clean up wait interrupted >>>>>> .... >>>>>> [continue with del] >>>>> >>>>> Yup, did this in #1. >>>>> >>>>>> >>>>>> 3) (minor) When there is a cleanruv task running and you run >>>>>> "ipa-replica-manage del", there is a unexpected error message with >>>>>> duplicate >>>>>> task object in LDAP: >>>>>> >>>>>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com --force >>>>>> Unable to connect to replica vm-072.idm.lab.bos.redhat.com, forcing >>>>>> removal >>>>>> FAIL >>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>>> contact LDAP server"} >>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>> >>>>>> There were issues removing a connection: This entry already exists >>>>>> <<<<<<<<< >>>>>> >>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>>> contact LDAP server"} >>>>>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record >>>>>> does not >>>>>> contain 'vm-072.idm.lab.bos.redhat.com.' >>>>>> You may need to manually remove them from the tree >>>>>> >>>>>> >>>>>> I think it should be enough to just catch for "entry already exists" in >>>>>> cleanallruv function, and in such case print a relevant error message >>>>>> bail out. >>>>>> Thus, self.conn.checkTask(dn, dowait=True) would not be called too. >>>>> >>>>> Good catch, fixed. >>>>> >>>>>> >>>>>> >>>>>> 4) (minor): In make_readonly function, there is a redundant "pass" >>>>>> statement: >>>>>> >>>>>> + def make_readonly(self): >>>>>> + """ >>>>>> + Make the current replication agreement read-only. >>>>>> + """ >>>>>> + dn = DN(('cn', 'userRoot'), ('cn', 'ldbm database'), >>>>>> + ('cn', 'plugins'), ('cn', 'config')) >>>>>> + >>>>>> + mod = [(ldap.MOD_REPLACE, 'nsslapd-readonly', 'on')] >>>>>> + try: >>>>>> + self.conn.modify_s(dn, mod) >>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>> + # We can't make the server we're removing read-only but >>>>>> + # this isn't a show-stopper >>>>>> + root_logger.debug("No permission to switch replica to >>>>>> read-only, >>>>>> continuing anyway") >>>>>> + pass <<<<<<<<<<<<<<< >>>>> >>>>> Yeah, this is one of my common mistakes. I put in a pass initially, then >>>>> add logging in front of it and forget to delete the pass. Its gone now. >>>>> >>>>>> >>>>>> >>>>>> 5) In clean_ruv, I think allowing a --force option to bypass the >>>>>> user_input >>>>>> would be helpful (at least for test automation): >>>>>> >>>>>> + if not ipautil.user_input("Continue to clean?", False): >>>>>> + sys.exit("Aborted") >>>>> >>>>> Yup, added. >>>>> >>>>> rob >>>> >>>> Slightly revised patch. I still had a window open with one unsaved change. >>>> >>>> rob >>>> >>> >>> Apparently there were two unsaved changes, one of which was lost. This adds in >>> the 'entry already exists' fix. >>> >>> rob >>> >> >> Just one last thing (otherwise the patch is OK) - I don't think this is what we >> want :-) >> >> # ipa-replica-manage clean-ruv 8 >> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 >> >> Cleaning the wrong replica ID will cause that server to no >> longer replicate so it may miss updates while the process >> is running. It would need to be re-initialized to maintain >> consistency. Be very careful. >> Continue to clean? [no]: y <<<<<< >> Aborted >> >> >> Nor this exception, (your are checking for wrong exception): >> >> # ipa-replica-manage clean-ruv 8 >> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 >> >> Cleaning the wrong replica ID will cause that server to no >> longer replicate so it may miss updates while the process >> is running. It would need to be re-initialized to maintain >> consistency. Be very careful. >> Continue to clean? [no]: >> unexpected error: This entry already exists >> >> This is the exception: >> >> Traceback (most recent call last): >> File "/sbin/ipa-replica-manage", line 651, in >> main() >> File "/sbin/ipa-replica-manage", line 648, in main >> clean_ruv(realm, args[1], options) >> File "/sbin/ipa-replica-manage", line 373, in clean_ruv >> thisrepl.cleanallruv(ruv) >> File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >> line 1136, in cleanallruv >> self.conn.addEntry(e) >> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 503, in >> addEntry >> self.__handle_errors(e, arg_desc=arg_desc) >> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 321, in >> __handle_errors >> raise errors.DuplicateEntry() >> ipalib.errors.DuplicateEntry: This entry already exists >> >> Martin >> > > Fixed that and a couple of other problems. When doing a disconnect we should > not also call clean-ruv. Ah, good self-catch. > > I also got tired of seeing crappy error messages so I added a little convert > utility. > > rob 1) There is CLEANALLRUV stuff included in 1050-3 and not here. There are also some finding for this new code. 2) We may want to bump Requires to higher version of 389-ds-base (389-ds-base-1.2.11.14-1) - it contains a fix for CLEANALLRUV+winsync bug I found earlier. 3) I just discovered another suspicious behavior. When we are deleting a master that has links also to other master(s) we delete those too. But we also automatically run CLEANALLRUV in these cases, so we may end up in multiple tasks being started on different masters - this does not look right. I think we may rather want to at first delete all links and then run CLEANALLRUV task, just for one time. This is what I get with current code: # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com Directory Manager password: Deleting a master is irreversible. To reconnect to the remote master you will need to prepare a new replica file and re-install. Continue to delete? [no]: yes ipa: INFO: Setting agreement cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping tree,cn=config schedule to 2358-2359 0 to force synch ipa: INFO: Deleting schedule 2358-2359 0 from agreement cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping tree,cn=config ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica acquired successfully: Incremental update succeeded: start: 0: end: 0 Background task created to clean replication data. This may take a while. This may be safely interrupted with Ctrl+C ^CWait for task interrupted. It will continue to run in the background Deleted replication agreement from 'vm-055.idm.lab.bos.redhat.com' to 'vm-072.idm.lab.bos.redhat.com' ipa: INFO: Setting agreement cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping tree,cn=config schedule to 2358-2359 0 to force synch ipa: INFO: Deleting schedule 2358-2359 0 from agreement cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping tree,cn=config ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica acquired successfully: Incremental update succeeded: start: 0: end: 0 Background task created to clean replication data. This may take a while. This may be safely interrupted with Ctrl+C ^CWait for task interrupted. It will continue to run in the background Deleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to 'vm-072.idm.lab.bos.redhat.com' Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record does not contain 'vm-072.idm.lab.bos.redhat.com.' You may need to manually remove them from the tree Martin From pviktori at redhat.com Fri Sep 14 12:53:37 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Fri, 14 Sep 2012 14:53:37 +0200 Subject: [Freeipa-devel] [PATCH] 0081 Only stop the main DS instance when upgrading it Message-ID: <505328D1.2020806@redhat.com> This fixes a 2.2?3.0 upgrade bug found while testing the Dogtag 10 work. See commit or ticket for details. https://fedorahosted.org/freeipa/ticket/3083 I also suspect that waiting for ports is not a good way to check if the CMS is fully initialized, but I don't know of a better way. If you know one, please speak up. -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0081-Only-stop-the-main-DS-instance-when-upgrading-it.patch Type: text/x-patch Size: 2481 bytes Desc: not available URL: From pviktori at redhat.com Fri Sep 14 12:59:31 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Fri, 14 Sep 2012 14:59:31 +0200 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <504F8EF8.2010601@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> <1347329065.24731.101.camel@aleeredhat.laptop> <504F318F.3030603@redhat.com> <504F35AD.3030905@redhat.com> <1347373691.24731.125.camel@aleeredhat.laptop> <504F4CE0.9080706@redhat.com> <504F67C5.4000600@redhat.com> <504F86C0.6060907@redhat.com> <504F8EF8.2010601@redhat.com> Message-ID: <50532A33.2050706@redhat.com> On 09/11/2012 09:20 PM, Rob Crittenden wrote: > Rob Crittenden wrote: >> Petr Viktorin wrote: >>> On 09/11/2012 04:38 PM, Rob Crittenden wrote: >>>> Ade Lee wrote: >>>>> On Tue, 2012-09-11 at 08:59 -0400, Rob Crittenden wrote: >>>>>> Petr Viktorin wrote: >>>>>>> On 09/11/2012 04:04 AM, Ade Lee wrote: >>>>>>>> On Mon, 2012-09-10 at 16:58 -0400, Rob Crittenden wrote: >>>>>>>>> Petr Viktorin wrote: >>>>>>>>>> Attaching rebased and squashed patches. I've done some testing >>>>>>>>>> with >>>>>>>>>> them >>>>>>>>>> but please test some more. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Most of these aren't IPA issues, but dogtag issues. I'll try to >>>>>>>>> split >>>>>>>>> them out. >>>>>>>>> >>>>>>>>> IPA: >>>>>>>>> >>>>>>>>> For the configuration files in install/conf to be updated at rpm >>>>>>>>> update >>>>>>>>> time the VERSION needs to be incremented. >>>>>>> >>>>>>> These files should stay the same since on upgrade we're still >>>>>>> using a >>>>>>> Dogtag 9 style instance. The Dogtag 10 ports are only used in new >>>>>>> installs. >>>>>>> >>>>>>>>> The ipa package lacks any updated dogtag dependencies, so I abused >>>>>>>>> it. >>>>>>> >>>>>>> What should the updated dependencies be? Since it should work with >>>>>>> both >>>>>>> dogtag 9 and 10, I don't see how they should change. >>>>>> >>>>>> I don't know either, but we need to prevent people from installing >>>>>> incompatible package combinations. >>>>>> >>>>> Would'nt the Conflicts: ipa < 3.0 in pki-ca mentioned below satisfy >>>>> this >>>>> requirement? The main concern is that you must have ipa 3.0 if you >>>>> have >>>>> dogtag 10. >>>>> >>>>> Given that dogtag is consumed by IPA though, it makes more sense to >>>>> put >>>>> the relevant conflicts in IPA rather than in dogtag. So in this case, >>>>> that would mean putting Conflicts: pki-ca >= 10.0 in IPA 2.x. >>>>> Recall that dogtag 10 will only be officially available in f18+. >>>> >>>> That isn't enough. If a F-17 user with IPA 2.2 installed upgrades to >>>> F-18 they would be able to install dogtag 10 and blow up their IPA >>>> server. >>>> >>>>> >>>>>>>>> I installed IPA with dogtag 9 and created a replica. >>>>>>>>> >>>>>>>>> I updated the IPA bits, that worked fine. >>>>>>>>> >>>>>>>>> I updated to dogtag 10 and now the CA doesn't work on the master, >>>>>>>>> including starting the dogtag instance. Note that the rpm update >>>>>>>>> process >>>>>>>>> worked, no notice that the CA service didn't restart. >>>>>>>>> >>>>>>>> Did you try to restart the CA with selinux in permissive mode? >>>>>>>> This is >>>>>>>> still required right now until I get the selinux policy all >>>>>>>> straightened >>>>>>>> out. >>>>>>>> >>>>>>>> There is also a separate dogtag ticket (which is currently being >>>>>>>> worked >>>>>>>> on) to restart existing dogtag instances when dogtag is upgraded >>>>>>>> from >>>>>>>> 9->10. >>>>>>> >>>>>>> In permissive mode, this upgrade works for me. >>>>>> >>>>>> I was in enforcing mode but saw no AVCs. What is the ETA on fixing >>>>>> this? >>>>>> >>>>> >>>>> Within the next week or two, I need to finish the IPA merge database >>>>> patch first, and then co-ordinate changes with the selinux guys. >>>>> >>>>>>> >>>>>>> >>>>>>> Sometimes I do get this error intermittently: >>>>>>> >>>>>>> ipa: ERROR: Certificate operation cannot be completed: Unable to >>>>>>> communicate with CMS (Service Temporarily Unavailable) >>>>>>> >>>>>>> Usually, waiting a couple of minutes clears this up. Perhaps we are >>>>>>> not >>>>>>> doing startup detection correctly. Ade mentioned that waiting for >>>>>>> ports >>>>>>> may not be ideal. How do we know if Dogtag is initialized? >>>>>> >>>>>> Years ago we had discussed with Andrew and Matt creating a URI that >>>>>> can >>>>>> be queried to determine dogtag status. I don't think that ever went >>>>>> anywhere. >>>>>> >>>>> Petr, this happens only on reboot, right? And not on regular "service >>>>> ipa restart"? >>> >>> I've now seen it happen right after a 9 ? 10 upgrade. >>> >>>>> Yeah, I remember this conversation - and even created a bug for it at >>>>> some point. This went away because the mechanism you were using >>>>> seemed >>>>> to be working. The timing may be very different now with tomcat 7 and >>>>> systemd. I'll open a dogtag trac ticket for this. >>>> >>>> Ok. >>>> >>>>> >>>>>>> >>>>>>>>> Uninstalling failed because it tried to run pkidestroy and not >>>>>>>>> pkiremove. >>>>>>> >>>>>>> I was under the impression that pkidestroy was the correct >>>>>>> command to >>>>>>> remove an upgraded instance. I'll check with Ade. >>>>>>> >>>>>>>> I'll test this too. >>>>>>>> >>>>>>>>> The contents of the file passed to pkispawn should be logged so we >>>>>>>>> can >>>>>>>>> see exactly what was passed in. >>>>>>>>> >>>>>>>> Its a pretty big file. You might want to only log the >>>>>>>> modifications. >>>>>>>> Or save the file somewhere. >>>>>>> >>>>>>> Our logs are pretty verbose, so that shouldn't be a problem. I'll >>>>>>> put it >>>>>>> in the next version of the patch. >>>>>> >>>>>> The question to ask is: would you need the contents of this file if >>>>>> all >>>>>> you got were logs and needed to evaluate why installation failed? In >>>>>> most cases this is yes. >>>>>> >>>>> Up to you guys. There is a patch I am working on in which I will be >>>>> logging the object that is being passed to the server from pkispawn. >>>>> That - and the diffs to the standard config file as I mentioned >>>>> above - >>>>> will likely be sufficient to debug almost all cases. >>>>> >>>>> Make sure not to log any passwords. >>>>> >>> >>> Thanks for the catch. Attaching updated patch that sanitizes the >>> passwords. >>> >>>>>>>>> DOGTAG: >>>>>>>>> >>>>>>>>> When upgrading using the dogtag-devel repo I had to specify >>>>>>>>> pki-tools.x86_64 otherwise it tried to install both 32 and 64-bit >>>>>>>>> versions (and failed). >>>>>>>>> >>>>>>>>> I ended up running: yum update pki-ca tomcatjss pki-tools.x86_64 >>>>>>>>> --enablerepo=dogtag-devel --enablerepo=updates-testing >>>>>>>>> >>>>>>>> We'll look into this. I think I know why this happens. >>>>>>>> >>>>>>>>> What happens if someone manually upgrades pki-ca without first >>>>>>>>> updating >>>>>>>>> ipa? I think that pki-ca is going to need a Conflicts ipa < 3.0 in >>>>>>>>> it. >>>>>>>> >>>>>>>> We can add that. >>>>>>>> >>>>>>>>> certificate renewal failed. I spent far too long trying to figure >>>>>>>>> out >>>>>>>>> why tomcat wasn't listening on port 9180 but failed. I think >>>>>>>>> 9180 is >>>>>>>>> actually the old server, right? So another missing dependency on a >>>>>>>>> fixed >>>>>>>>> certmonger? >>>>>>>>> >>>>>>>>> The best I could find was the certmonger error: >>>>>>>>> >>>>>>>>> ca-error: Error 7 connecting to >>>>>>>>> http://edsel.example.com:9180/ca/ee/ca/profileSubmit: Couldn't >>>>>>>>> connect >>>>>>>>> to server. >>>>>>>>> >>>>>>> >>>>>>> I'll set the certmonger min version to 0.60 as per Nalin's mail. >>>>>> >>>>>> Ok. >>>>>> >>>>>>>> Is this cert renewal on a dogtag 10 instance? Or the upgraded >>>>>>>> dogtag 9? >>>>>>>> If its dogtag 10, perhaps you do not have the certmonger version >>>>>>>> that >>>>>>>> has the relevant fix? If its dogtag 9, then we need to figure out >>>>>>>> whats >>>>>>>> going on. That reminds me - I need to file a bug to allow >>>>>>>> certmonger to >>>>>>>> talk to the newly defined dogtag ports. Do you have selinux >>>>>>>> permissive? >>>>>>>> >>>>>>>>> There is no man page for pkispawn/pkidestroy :-( According to the >>>>>>>>> FHS >>>>>>>>> these should not be in /bin but in /usr/sbin (not end-user >>>>>>>>> commands). >>>>>>>>> >>>>>>>> There is a trac ticket open for man pages for pkispawn and >>>>>>>> pkidestroy. >>>>>>>> We plan to complete this ticket by the time f18 is released. >>>>>>>> >>>>>>>> We'll look into the location of pkispawn/pkicreate. >>>>>>>> >>>>>>>>> The output of pkicreate/pkisilent was really terrible and not >>>>>>>>> usable at >>>>>>>>> all so we didn't display it when failures occurred. It looks like >>>>>>>>> that >>>>>>>>> has been addressed, at least for the case where a CA is already >>>>>>>>> configured and you try to install IPA. Perhaps we should capture >>>>>>>>> stderr >>>>>>>>> and display that instead of the command-line of pkispawn? Again, a >>>>>>>>> man >>>>>>>>> page would help with the integration. >>>>>>>>> >>>>>>>>> 2012-09-10T20:51:45Z DEBUG [2/18]: configuring certificate >>>>>>>>> server >>>>>>>>> instance >>>>>>>>> 2012-09-10T20:51:45Z DEBUG args=/bin/pkispawn -s CA -f >>>>>>>>> /tmp/tmp_Urraq >>>>>>>>> 2012-09-10T20:51:45Z DEBUG stdout= >>>>>>>>> 2012-09-10T20:51:45Z DEBUG stderr=pkispawn : ERROR ....... >>>>>>>>> PKI >>>>>>>>> subsystem 'CA' for instance 'pki-tomcat' already exists! >>>>>>>>> >>>>>>>>> 2012-09-10T20:51:45Z CRITICAL failed to configure ca instance >>>>>>>>> Command >>>>>>>>> '/bin/pkispawn -s CA -f /tmp/tmp_Urraq' returned non-zero exit >>>>>>>>> status 1 >>>>>>>>> >>>>>>>> That may be a good idea. Of course. thats an IPA thing, right? >>>>>> >>>>>> Right, not a show-stopper for this but a new ticket should be >>>>>> opened to >>>>>> track it. >>> >>> https://fedorahosted.org/freeipa/ticket/3072 >> >> Thanks. >> >> I tested a 2.2 -> 3.0 upgrade with dogtag 9 and it seems to have failed >> to restart something: >> >> [ post rpm -Uvh dist/rpms/freeipa*.rpm ] >> >> [root at edsel freeipa]# ipa cert-show 1 >> ipa: ERROR: Certificate operation cannot be completed: Unable to >> communicate with CMS (Service Temporarily Unavailable) >> [root at edsel freeipa]# ipactl restart >> Restarting Directory Service >> Restarting KDC Service >> Restarting KPASSWD Service >> Restarting MEMCACHE Service >> Restarting HTTP Service >> Restarting CA Service >> [root at edsel freeipa]# ipa cert-show 1 >> Certificate: MIIDjTCCAnWgAwIBAgIBATANBgk >> ... >> >> The apache log had this: >> >> [Tue Sep 11 14:35:42 2012] [error] (111)Connection refused: proxy: AJP: >> attempt to connect to 127.0.0.1:9447 (localhost) failed >> [Tue Sep 11 14:35:42 2012] [error] ap_proxy_connect_backend disabling >> worker for (localhost) >> [Tue Sep 11 14:35:42 2012] [error] proxy: AJP: failed to make connection >> to backend: localhost >> >> So I'm gathering that dogtag didn't restart properly, but I'm just >> guessing. It could have been the PKI-IPA ds instance too, I'm not sure >> where to check. >> I believe I found the cause here. See my patch 81 or https://fedorahosted.org/freeipa/ticket/3083. Thanks to Ade for helping me make sense of the logs. >> I also noticed this in /var/log/ipaupgrade.log: >> >> 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl start certmonger.service >> 2012-09-11T18:28:22Z DEBUG stdout= >> 2012-09-11T18:28:22Z DEBUG stderr= >> 2012-09-11T18:28:22Z DEBUG args=/bin/systemctl is-active >> certmonger.service >> 2012-09-11T18:28:22Z DEBUG stdout=active >> ... >> ... [ snip certutil output ] >> ... >> 2012-09-11T18:28:52Z DEBUG args=/usr/bin/getcert start-tracking -d >> /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c >> dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert >> "auditSigningCert cert-pki-ca" -P XXXXXXXX >> 2012-09-11T18:28:52Z DEBUG stdout=Please verify that the certmonger >> service is still running. >> >> 2012-09-11T18:28:52Z DEBUG stderr= >> 2012-09-11T18:28:52Z INFO File >> "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", >> line 614, in run_script >> return_value = main_function() >> >> File "/usr/sbin/ipa-upgradeconfig", line 540, in main >> enable_certificate_renewal(api.env.realm) >> >> File "/usr/sbin/ipa-upgradeconfig", line 455, in >> enable_certificate_renewal >> ca.configure_renewal() >> >> File >> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line >> 1298, in configure_renewal >> self.dogtag_constants.ALIAS_DIR, 'renew_ca_cert "%s"' % nickname) >> >> File "/usr/lib/python2.7/site-packages/ipapython/certmonger.py", line >> 394, in dogtag_start_tracking >> (stdout, stderr, returncode) = ipautil.run(args, nolog=[pin]) >> >> File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line >> 309, in run >> raise CalledProcessError(p.returncode, args) >> >> 2012-09-11T18:28:52Z INFO The ipa-upgradeconfig command failed, >> exception: CalledProcessError: Command '/usr/bin/getcert start-tracking >> -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c >> dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ca_cert >> "auditSigningCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit >> status 1 >> >> I'm not sure if this is related to this patch or not. If it isn't can >> you file a new ticket on it, or add it the 2.2 -> 3.0 upgrade ticket? > > These are both easily reproducable. Rather than restarting the world > this time I looked to see what was running: > > # ipactl status > Directory Service: RUNNING > KDC Service: RUNNING > KPASSWD Service: RUNNING > MEMCACHE Service: RUNNING > HTTP Service: RUNNING > CA Service: RUNNING > > # ps -ef|grep java > > > # /bin/systemctl -a status pki-cad at pki-ca.service > pki-cad at pki-ca.service - PKI Certificate Authority Server pki-ca > Loaded: loaded (/usr/lib/systemd/system/pki-cad at .service; > enabled) > Active: failed (Result: exit-code) since Tue, 11 Sep 2012 > 15:10:35 -0400; 8min ago > Process: 5917 ExecStop=/usr/bin/pkicontrol stop ca %i > (code=exited, status=1/FAILURE) > Process: 5763 ExecStart=/usr/bin/pkicontrol start ca %i > (code=exited, status=0/SUCCESS) > Main PID: 5830 (code=exited, status=0/SUCCESS) > CGroup: name=systemd:/system/pki-cad at .service/pki-ca > > Sep 11 15:10:27 edsel.greyoak.com runuser[5792]: > pam_unix(runuser-l:session): session opened for user pkiuser by (uid=0) > Sep 11 15:10:27 edsel.greyoak.com runuser[5792]: > pam_unix(runuser-l:session): session closed for user pkiuser > Sep 11 15:10:35 edsel.greyoak.com runuser[5936]: > pam_unix(runuser-l:session): session opened for user pkiuser by (uid=0) > Sep 11 15:10:35 edsel.greyoak.com runuser[5936]: > pam_unix(runuser-l:session): session closed for user pkiuser > > So I'm thinking that IPA believes that the CA is running, and systemd > reported it started, but something went wrong. > > The catalina.out log doesn't really show a failure, just the stupid "I > can't stop because I'm already stopped" traceback. I wonder if this is > an issue with pkicontrol where restart doesn't handle a stop failure, > but I'm just guessing. > > I suspect that the start-tracking issue is not related. > > rob -- Petr? From pvoborni at redhat.com Fri Sep 14 13:00:59 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Fri, 14 Sep 2012 15:00:59 +0200 Subject: [Freeipa-devel] [PATCH] 214 Fix jquery error when using '??' in a pkey Message-ID: <50532A8B.2060309@redhat.com> This patch is only for FreeIPA 2.2. It is already fixed in 3.0. If '??' is used in a adder dialog as a pkey it can cause "jQuery15208158273949015573_1346241267446 was not called" error. Update of jquery library fixes the issue. Update reveals an incorrect handler definition issue in ssh_key_widget, which is also fixed. https://bugzilla.redhat.com/show_bug.cgi?id=855278 https://fedorahosted.org/freeipa/ticket/3073 -- Petr Vobornik -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pvoborni-0214-Fix-jquery-error-when-using-in-a-pkey.patch Type: text/x-patch Size: 323960 bytes Desc: not available URL: From atkac at redhat.com Fri Sep 14 13:07:57 2012 From: atkac at redhat.com (Adam Tkac) Date: Fri, 14 Sep 2012 15:07:57 +0200 Subject: [Freeipa-devel] [PATCH 0055] Fix race condition in addrdataset() during SOA serial update In-Reply-To: <5049D501.20207@redhat.com> References: <5049D501.20207@redhat.com> Message-ID: <20120914130756.GA10160@redhat.com> On Fri, Sep 07, 2012 at 01:05:37PM +0200, Petr Spacek wrote: > Hello, > > Fix race condition in addrdataset() during SOA serial update. > > https://fedorahosted.org/bind-dyndb-ldap/ticket/89 Good catch, thanks. Ack A > From 5e8bc8f943345d8d92900474905288939958dcd8 Mon Sep 17 00:00:00 2001 > From: Petr Spacek > Date: Fri, 7 Sep 2012 13:01:57 +0200 > Subject: [PATCH] Fix race condition in addrdataset() during SOA serial > update. > > https://fedorahosted.org/bind-dyndb-ldap/ticket/89 > > Signed-off-by: Petr Spacek > --- > src/ldap_driver.c | 44 ++++++++++++++++++++++++++++++++++---------- > src/ldap_helper.c | 4 ++-- > 2 files changed, 36 insertions(+), 12 deletions(-) > > diff --git a/src/ldap_driver.c b/src/ldap_driver.c > index 2cdde30cdad9544d530475f5cf4a0b8275a56f03..3a802238028145d35390f6a8d00f156bfdf8e7a1 100644 > --- a/src/ldap_driver.c > +++ b/src/ldap_driver.c > @@ -936,6 +936,7 @@ addrdataset(dns_db_t *db, dns_dbnode_t *node, dns_dbversion_t *version, > dns_rdatalist_t diff; > isc_result_t result; > isc_boolean_t rdatalist_exists = ISC_FALSE; > + isc_boolean_t soa_simulated_write = ISC_FALSE; > > UNUSED(now); > UNUSED(db); > @@ -975,42 +976,65 @@ addrdataset(dns_db_t *db, dns_dbnode_t *node, dns_dbversion_t *version, > rdatalist_removedups(found_rdlist, new_rdlist, > ISC_FALSE, &diff); > > - if ((options & DNS_DBADD_MERGE) != 0) > + if ((options & DNS_DBADD_MERGE) == 0 && > + (rdatalist_length(&diff) != 0)) { > + CLEANUP_WITH(DNS_R_NOTEXACT); > + } else { > free_rdatalist(ldapdb->common.mctx, &diff); > - else if (rdatalist_length(&diff) != 0) { > - free_rdatalist(ldapdb->common.mctx, &diff); > - result = DNS_R_NOTEXACT; > - goto cleanup; > } > } else { > /* Replace existing rdataset */ > free_rdatalist(ldapdb->common.mctx, found_rdlist); > } > } > > - CHECK(write_to_ldap(&ldapdbnode->owner, ldapdb->ldap_inst, new_rdlist)); > + /* HACK: SOA addition will never fail with DNS_R_UNCHANGED. > + * This prevents warning from BIND's diff_apply(), it has too strict > + * checks for us. > + * > + * Reason: There is a race condition between SOA serial update > + * from BIND's update_action() and our persistent search watcher, because > + * they don't know about each other. > + * BIND's update_action() changes data with first addrdataset() call and > + * then changes serial with second addrdataset() call. > + * It can lead to empty diff if persistent search watcher > + * incremented serial in meanwhile. > + */ > + if (HEAD(new_rdlist->rdata) == NULL) { > + if (rdlist->type == dns_rdatatype_soa) > + soa_simulated_write = ISC_TRUE; > + else > + CLEANUP_WITH(DNS_R_UNCHANGED); > + } else { > + CHECK(write_to_ldap(&ldapdbnode->owner, ldapdb->ldap_inst, new_rdlist)); > + } > + > > if (addedrdataset != NULL) { > - result = dns_rdatalist_tordataset(new_rdlist, addedrdataset); > - /* Use strong condition here, returns only SUCCESS */ > - INSIST(result == ISC_R_SUCCESS); > + if (soa_simulated_write) { > + dns_rdataset_clone(rdataset, addedrdataset); > + } else { > + result = dns_rdatalist_tordataset(new_rdlist, addedrdataset); > + /* Use strong condition here, returns only SUCCESS */ > + INSIST(result == ISC_R_SUCCESS); > + } > } > > if (rdatalist_exists) { > ISC_LIST_APPENDLIST(found_rdlist->rdata, new_rdlist->rdata, > link); > SAFE_MEM_PUT_PTR(ldapdb->common.mctx, new_rdlist); > } else > APPEND(ldapdbnode->rdatalist, new_rdlist, link); > > - > return ISC_R_SUCCESS; > > cleanup: > if (new_rdlist != NULL) { > free_rdatalist(ldapdb->common.mctx, new_rdlist); > SAFE_MEM_PUT_PTR(ldapdb->common.mctx, new_rdlist); > } > + free_rdatalist(ldapdb->common.mctx, &diff); > > return result; > } > diff --git a/src/ldap_helper.c b/src/ldap_helper.c > index 3241ffe486205fa03a6fd1a0a14edf1245c5c4aa..e636a84b35d0bcdc8573c6e7146f38ee21a42076 100644 > --- a/src/ldap_helper.c > +++ b/src/ldap_helper.c > @@ -2973,10 +2973,10 @@ soa_serial_increment(isc_mem_t *mctx, ldap_instance_t *inst, > > /* put the new SOA to inst->cache and compare old and new serials */ > CHECK(ldap_get_zone_serial(inst, zone_name, &new_serial)); > - INSIST(isc_serial_gt(new_serial, old_serial) == ISC_TRUE); > > cleanup: > - if (result != ISC_R_SUCCESS) > + if (result != ISC_R_SUCCESS || > + isc_serial_gt(new_serial, old_serial) != ISC_TRUE) > log_error("SOA serial number incrementation failed in zone '%s'", > str_buf(zone_dn)); > > -- > 1.7.11.4 > -- Adam Tkac, Red Hat, Inc. From atkac at redhat.com Fri Sep 14 13:10:37 2012 From: atkac at redhat.com (Adam Tkac) Date: Fri, 14 Sep 2012 15:10:37 +0200 Subject: [Freeipa-devel] [PATCH 0056] Fix crash caused by zone deletion vs. SOA serial increment race condition In-Reply-To: <5050650B.3000709@redhat.com> References: <5050650B.3000709@redhat.com> Message-ID: <20120914131037.GB10160@redhat.com> On Wed, Sep 12, 2012 at 12:33:47PM +0200, Petr Spacek wrote: > Hello, > > The patch fixes crash caused by stupid bug in logging code. Ack. > From 01aa00f9ba4feac9f97b34b81c3697b2b7f8122f Mon Sep 17 00:00:00 2001 > From: Petr Spacek > Date: Fri, 7 Sep 2012 16:21:27 +0200 > Subject: [PATCH] Fix crash caused by zone deletion vs. SOA serial increment > race condition. > > Signed-off-by: Petr Spacek > --- > src/ldap_helper.c | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > diff --git a/src/ldap_helper.c b/src/ldap_helper.c > index e636a84b35d0bcdc8573c6e7146f38ee21a42076..058048f41485999be0d8ffeadea02f2e25879370 100644 > --- a/src/ldap_helper.c > +++ b/src/ldap_helper.c > @@ -2931,6 +2931,7 @@ soa_serial_increment(isc_mem_t *mctx, ldap_instance_t *inst, > dns_name_t *zone_name) { > isc_result_t result = ISC_R_FAILURE; > ld_string_t *zone_dn = NULL; > + const char *zone_dn_char = "INACTIVE/UNKNOWN"; > ldapdb_rdatalist_t rdatalist; > dns_rdatalist_t *rdlist = NULL; > dns_rdata_t *soa_rdata = NULL; > @@ -2944,6 +2945,7 @@ soa_serial_increment(isc_mem_t *mctx, ldap_instance_t *inst, > INIT_LIST(rdatalist); > CHECK(str_new(mctx, &zone_dn)); > CHECK(dnsname_to_dn(inst->zone_register, zone_name, zone_dn)); > + zone_dn_char = str_buf(zone_dn); > log_debug(5, "incrementing SOA serial number in zone '%s'", > str_buf(zone_dn)); > > @@ -2978,7 +2980,7 @@ cleanup: > if (result != ISC_R_SUCCESS || > isc_serial_gt(new_serial, old_serial) != ISC_TRUE) > log_error("SOA serial number incrementation failed in zone '%s'", > - str_buf(zone_dn)); > + zone_dn_char); > > str_destroy(&zone_dn); > ldapdb_rdatalist_destroy(mctx, &rdatalist); > -- > 1.7.11.4 > -- Adam Tkac, Red Hat, Inc. From simo at redhat.com Fri Sep 14 13:12:17 2012 From: simo at redhat.com (Simo Sorce) Date: Fri, 14 Sep 2012 09:12:17 -0400 Subject: [Freeipa-devel] [PATCH] 0081 Only stop the main DS instance when upgrading it In-Reply-To: <505328D1.2020806@redhat.com> References: <505328D1.2020806@redhat.com> Message-ID: <1347628337.19861.31.camel@willson.li.ssimo.org> On Fri, 2012-09-14 at 14:53 +0200, Petr Viktorin wrote: > This fixes a 2.2?3.0 upgrade bug found while testing the Dogtag 10 work. > See commit or ticket for details. > > https://fedorahosted.org/freeipa/ticket/3083 > > > > I also suspect that waiting for ports is not a good way to check if the > CMS is fully initialized, but I don't know of a better way. If you know > one, please speak up. > Won;t we get back to square zero when the work to merge DS and CS instances into one will be completed ? Simo. -- Simo Sorce * Red Hat, Inc * New York From jcholast at redhat.com Fri Sep 14 13:18:50 2012 From: jcholast at redhat.com (Jan Cholasta) Date: Fri, 14 Sep 2012 15:18:50 +0200 Subject: [Freeipa-devel] [PATCH] 309 Fix addattr internal error In-Reply-To: <5052D160.1000701@redhat.com> References: <5051EEB8.6050303@redhat.com> <505231C6.9060105@redhat.com> <5052D160.1000701@redhat.com> Message-ID: <50532EBA.6030706@redhat.com> Dne 14.9.2012 08:40, Martin Kosek napsal(a): > On 09/13/2012 09:19 PM, Rob Crittenden wrote: >> Martin Kosek wrote: >>> When ADD command is being executed and a single-value object attribute >>> is being set with both option and addattr IPA ends up in an internal >>> error. >>> >>> Make better value sanitizing job in this case and let IPA throw >>> a user-friendly error. Unit test exercising this situation is added. >>> >>> https://fedorahosted.org/freeipa/ticket/2429 >> >> + if not isinstance(val, (list, tuple)): >> + val = [val] >> + val.extend(adddict[attr]) >> >> I val is a tuple the extend is still going to fail. Can val ever be a tuple? If >> so we'd need to convert it to a list. >> >> rob > > I don't think it could be a tuple, I am about 99% certain. So for this 1% I > added a special clause for tuple. Patch attached. It is possible that it can be a tuple ATM. > > We will be able to be even more certain when Honza finishes his strict encoding > patch he was working on in the summer. With his patch, the attributes should > always be a list. Yes. Also, this was already fixed in my patch for , but it was reverted. > > Martin > Honza -- Jan Cholasta From pspacek at redhat.com Fri Sep 14 13:17:24 2012 From: pspacek at redhat.com (Petr Spacek) Date: Fri, 14 Sep 2012 15:17:24 +0200 Subject: [Freeipa-devel] [PATCH 0056] Fix crash caused by zone deletion vs. SOA serial increment race condition In-Reply-To: <20120914131037.GB10160@redhat.com> References: <5050650B.3000709@redhat.com> <20120914131037.GB10160@redhat.com> Message-ID: <50532E64.3030809@redhat.com> On 09/14/2012 03:10 PM, Adam Tkac wrote: > On Wed, Sep 12, 2012 at 12:33:47PM +0200, Petr Spacek wrote: >> Hello, >> >> The patch fixes crash caused by stupid bug in logging code. > > Ack. > Pushed to master: https://fedorahosted.org/bind-dyndb-ldap/changeset/da558d15329f6e2308ad69118545363b9adbd0d1 Petr^2 Spacek From pspacek at redhat.com Fri Sep 14 13:17:30 2012 From: pspacek at redhat.com (Petr Spacek) Date: Fri, 14 Sep 2012 15:17:30 +0200 Subject: [Freeipa-devel] [PATCH 0055] Fix race condition in addrdataset() during SOA serial update In-Reply-To: <20120914130756.GA10160@redhat.com> References: <5049D501.20207@redhat.com> <20120914130756.GA10160@redhat.com> Message-ID: <50532E6A.2060902@redhat.com> On 09/14/2012 03:07 PM, Adam Tkac wrote: > On Fri, Sep 07, 2012 at 01:05:37PM +0200, Petr Spacek wrote: >> Hello, >> >> Fix race condition in addrdataset() during SOA serial update. >> >> https://fedorahosted.org/bind-dyndb-ldap/ticket/89 > > Good catch, thanks. Ack > > A Pushed to master: https://fedorahosted.org/bind-dyndb-ldap/changeset/dca6558ae5224674878972dc1e71e846a13bf139 Petr^2 Spacek From atkac at redhat.com Fri Sep 14 13:23:34 2012 From: atkac at redhat.com (Adam Tkac) Date: Fri, 14 Sep 2012 15:23:34 +0200 Subject: [Freeipa-devel] [PATCH 0057] Fix LDAP operation selection logic in ldap_modify_do() In-Reply-To: <5050656D.8010604@redhat.com> References: <5050656D.8010604@redhat.com> Message-ID: <20120914132334.GC10160@redhat.com> On Wed, Sep 12, 2012 at 12:35:25PM +0200, Petr Spacek wrote: > Hello, > > There is a fix for LDAP operation selection logic in ldap_modify_do(). > > Each operation code in LDAPMod structure can be ORed > with LDAP_MOD_BVALUES. Ack > From ab11e62ec2496f2c7245c4d8d80c2fd189b68aa9 Mon Sep 17 00:00:00 2001 > From: Petr Spacek > Date: Tue, 11 Sep 2012 16:23:18 +0200 > Subject: [PATCH] Fix LDAP operation selection logic in ldap_modify_do(). > > Each operation code in LDAPMod structure can be ORed > with LDAP_MOD_BVALUES. > > Signed-off-by: Petr Spacek > --- > src/ldap_helper.c | 29 +++++++++++++++++------------ > 1 file changed, 17 insertions(+), 12 deletions(-) > > diff --git a/src/ldap_helper.c b/src/ldap_helper.c > index 058048f41485999be0d8ffeadea02f2e25879370..d9c7ce5d84c3944a86ff1865ff6be073ddc294c8 100644 > --- a/src/ldap_helper.c > +++ b/src/ldap_helper.c > @@ -2149,33 +2149,38 @@ ldap_modify_do(ldap_instance_t *ldap_inst, ldap_connection_t *ldap_conn, > CHECK(ldap_connect(ldap_inst, ldap_conn, ISC_FALSE)); > } > > + /* Any mod_op can be ORed with LDAP_MOD_BVALUES. */ > + if ((mods[0]->mod_op & ~LDAP_MOD_BVALUES) == LDAP_MOD_ADD) > + operation_str = "modifying(add)"; > + else if ((mods[0]->mod_op & ~LDAP_MOD_BVALUES) == LDAP_MOD_DELETE) > + operation_str = "modifying(del)"; > + else if ((mods[0]->mod_op & ~LDAP_MOD_BVALUES) == LDAP_MOD_REPLACE) > + operation_str = "modifying(replace)"; > + else { > + operation_str = "modifying(unknown operation)"; > + log_bug("%s: 0x%x", operation_str, mods[0]->mod_op); > + CHECK(ISC_R_NOTIMPLEMENTED); > + } > + > if (delete_node) { > log_debug(2, "deleting whole node: '%s'", dn); > ret = ldap_delete_ext_s(ldap_conn->handle, dn, NULL, NULL); > } else { > - log_debug(2, "writing to '%s'", dn); > + log_debug(2, "writing to '%s': %s", dn, operation_str); > ret = ldap_modify_ext_s(ldap_conn->handle, dn, mods, NULL, NULL); > } > > result = (ret == LDAP_SUCCESS) ? ISC_R_SUCCESS : ISC_R_FAILURE; > if (ret == LDAP_SUCCESS) > goto cleanup; > > - if (mods[0]->mod_op == LDAP_MOD_ADD) > - operation_str = "modifying(add)"; > - else if (mods[0]->mod_op == LDAP_MOD_DELETE) > - operation_str = "modifying(del)"; > - else { > - operation_str = "modifying(unknown operation)"; > - CHECK(ISC_R_NOTIMPLEMENTED); > - } > - > LDAP_OPT_CHECK(ldap_get_option(ldap_conn->handle, LDAP_OPT_RESULT_CODE, > &err_code), "ldap_modify_do(%s) failed to obtain ldap error code", > operation_str); > > /* If there is no object yet, create it with an ldap add operation. */ > - if (mods[0]->mod_op == LDAP_MOD_ADD && err_code == LDAP_NO_SUCH_OBJECT) { > + if ((mods[0]->mod_op & ~LDAP_MOD_BVALUES) == LDAP_MOD_ADD && > + err_code == LDAP_NO_SUCH_OBJECT) { > int i; > LDAPMod **new_mods; > char *obj_str[] = { "idnsRecord", NULL }; > @@ -2211,7 +2216,7 @@ ldap_modify_do(ldap_instance_t *ldap_inst, ldap_connection_t *ldap_conn, > > /* do not error out if we are trying to delete an > * unexisting attribute */ > - if (mods[0]->mod_op != LDAP_MOD_DELETE || > + if ((mods[0]->mod_op & ~LDAP_MOD_BVALUES) != LDAP_MOD_DELETE || > err_code != LDAP_NO_SUCH_ATTRIBUTE) { > result = ISC_R_FAILURE; > } > -- > 1.7.11.4 > -- Adam Tkac, Red Hat, Inc. From atkac at redhat.com Fri Sep 14 13:24:24 2012 From: atkac at redhat.com (Adam Tkac) Date: Fri, 14 Sep 2012 15:24:24 +0200 Subject: [Freeipa-devel] [PATCH 0058] Improve persistent search logging In-Reply-To: <505065B6.50004@redhat.com> References: <505065B6.50004@redhat.com> Message-ID: <20120914132423.GD10160@redhat.com> On Wed, Sep 12, 2012 at 12:36:38PM +0200, Petr Spacek wrote: > Hello, > > this patch adds "result" codes to error messages in persistent search code. Ack. > From f6cb53278d8f39ac6da4fb8e26820f6ee02ae6e3 Mon Sep 17 00:00:00 2001 > From: Petr Spacek > Date: Wed, 12 Sep 2012 12:27:51 +0200 > Subject: [PATCH] Improve persistent search logging. > > Signed-off-by: Petr Spacek > --- > src/ldap_helper.c | 14 +++++++------- > 1 file changed, 7 insertions(+), 7 deletions(-) > > diff --git a/src/ldap_helper.c b/src/ldap_helper.c > index d9c7ce5d84c3944a86ff1865ff6be073ddc294c8..92edbe7159272772e1c993d46da7c93382cbc5d4 100644 > --- a/src/ldap_helper.c > +++ b/src/ldap_helper.c > @@ -3069,9 +3069,9 @@ update_zone(isc_task_t *task, isc_event_t *event) > > cleanup: > if (result != ISC_R_SUCCESS) > - log_error("update_action (psearch) failed for '%s': %s. " > + log_error_r("update_zone (psearch) failed for '%s'. " > "Zones can be outdated, run `rndc reload`", > - pevent->dn, isc_result_totext(result)); > + pevent->dn); > > ldap_query_free(ISC_FALSE, &ldap_qresult_zone); > ldap_query_free(ISC_FALSE, &ldap_qresult_record); > @@ -3125,7 +3125,7 @@ update_config(isc_task_t *task, isc_event_t *event) > > cleanup: > if (result != ISC_R_SUCCESS) > - log_error("update_config (psearch) failed for %s. " > + log_error_r("update_config (psearch) failed for '%s'. " > "Configuration can be outdated, run `rndc reload`", > pevent->dn); > > @@ -3221,9 +3221,9 @@ update_record(isc_task_t *task, isc_event_t *event) > } > cleanup: > if (result != ISC_R_SUCCESS) > - log_error("update_record (psearch) failed, dn '%s'. " > + log_error_r("update_record (psearch) failed, dn '%s' change type 0x%x. " > "Records can be outdated, run `rndc reload`", > - pevent->dn); > + pevent->dn, pevent->chgtype); > > if (dns_name_dynamic(&name)) > dns_name_free(&name, inst->mctx); > @@ -3400,7 +3400,7 @@ cleanup: > if (prevdn_ldap != NULL) > ldap_memfree(prevdn); > > - log_error("psearch_update failed for %s zone. " > + log_error_r("psearch_update failed for '%s' zone. " > "Zone can be outdated, run `rndc reload`", > entry->dn); > } > @@ -3586,7 +3586,7 @@ restart: > * Error means inconsistency of our zones > * data. > */ > - log_error("ldap_psearch_watcher failed, zones " > + log_error_r("ldap_psearch_watcher failed, zones " > "might be outdated. Run `rndc reload`"); > goto soft_err; > } > -- > 1.7.11.4 > -- Adam Tkac, Red Hat, Inc. From atkac at redhat.com Fri Sep 14 13:34:13 2012 From: atkac at redhat.com (Adam Tkac) Date: Fri, 14 Sep 2012 15:34:13 +0200 Subject: [Freeipa-devel] [PATCH 0059] Fix potential crash after free(uninitialized variable) In-Reply-To: <50506D0C.70706@redhat.com> References: <50506D0C.70706@redhat.com> Message-ID: <20120914133412.GA10557@redhat.com> On Wed, Sep 12, 2012 at 01:07:56PM +0200, Petr Spacek wrote: > Hello, > > This patch fixes potential crash after free(uninitialized variable) > in persistent search code. > > Coverity CID 13088. Ack > From 3197b4ace3e852495bf4f9fdc32192459160027c Mon Sep 17 00:00:00 2001 > From: Petr Spacek > Date: Wed, 12 Sep 2012 13:04:39 +0200 > Subject: [PATCH] Fix potential crash after free(uninitialized variable) in > persistent search code. > > Coverity CID 13088. > > Signed-off-by: Petr Spacek > --- > src/ldap_helper.c | 15 +++++++-------- > 1 file changed, 7 insertions(+), 8 deletions(-) > > diff --git a/src/ldap_helper.c b/src/ldap_helper.c > index 92edbe7159272772e1c993d46da7c93382cbc5d4..67a64b79159983c83cb1bfc73c4b02a9bce986a7 100644 > --- a/src/ldap_helper.c > +++ b/src/ldap_helper.c > @@ -2878,8 +2878,8 @@ cleanup: > static isc_result_t > ldap_pscontrol_create(LDAPControl **ctrlp) > { > - BerElement *ber; > - struct berval *berval; > + BerElement *ber = NULL; > + struct berval *berval = NULL; > isc_result_t result = ISC_R_FAILURE; > > REQUIRE(ctrlp != NULL && *ctrlp == NULL); > @@ -2905,14 +2905,13 @@ ldap_pscontrol_create(LDAPControl **ctrlp) > != LDAP_SUCCESS) > goto cleanup; > > - ber_free(ber, 1); > - ber_bvfree(berval); > - > - return ISC_R_SUCCESS; > + result = ISC_R_SUCCESS; > > cleanup: > - ber_free(ber, 1); > - ber_bvfree(berval); > + if (ber != NULL) > + ber_free(ber, 1); > + if (berval != NULL) > + ber_bvfree(berval); > > return result; > } > -- > 1.7.11.4 > -- Adam Tkac, Red Hat, Inc. From pviktori at redhat.com Fri Sep 14 13:45:49 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Fri, 14 Sep 2012 15:45:49 +0200 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <5050BB14.3050502@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <1345038855.21349.144.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> <1347329065.24731.101.camel@aleeredhat.laptop> <504F318F.3030603@redhat.com> <504F35AD.3030905@redhat.com> <1347373691.24731.125.camel@aleeredhat.laptop> <504F4CE0.9080706@redhat.com> <504F67C5.4000600@redhat.com> <504F86C0.6060907@redhat.com> <5050BB14.3050502@redhat.com> Message-ID: <5053350D.5010805@redhat.com> On 09/12/2012 06:40 PM, Petr Viktorin wrote: > A new Dogtag build with changed pkispawn/pkidestroy locations should be > out later today. The attached patch should work with that build. > One more update: the wrong constant was used in ipa-upgradeconfig on a d9-style instance under d10. -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0074-08-Use-Dogtag-10-only-when-it-is-available.patch Type: text/x-patch Size: 63144 bytes Desc: not available URL: From pspacek at redhat.com Fri Sep 14 13:57:54 2012 From: pspacek at redhat.com (Petr Spacek) Date: Fri, 14 Sep 2012 15:57:54 +0200 Subject: [Freeipa-devel] [PATCH 0057] Fix LDAP operation selection logic in ldap_modify_do() In-Reply-To: <20120914132334.GC10160@redhat.com> References: <5050656D.8010604@redhat.com> <20120914132334.GC10160@redhat.com> Message-ID: <505337E2.7060804@redhat.com> On 09/14/2012 03:23 PM, Adam Tkac wrote: > On Wed, Sep 12, 2012 at 12:35:25PM +0200, Petr Spacek wrote: >> Hello, >> >> There is a fix for LDAP operation selection logic in ldap_modify_do(). >> >> Each operation code in LDAPMod structure can be ORed >> with LDAP_MOD_BVALUES. > > Ack Pushed to master: https://fedorahosted.org/bind-dyndb-ldap/changeset/ff9f644ec02b8900cb3bdaed91023f6a10983e28 Petr^2 Spacek From pspacek at redhat.com Fri Sep 14 13:58:40 2012 From: pspacek at redhat.com (Petr Spacek) Date: Fri, 14 Sep 2012 15:58:40 +0200 Subject: [Freeipa-devel] [PATCH 0058] Improve persistent search logging In-Reply-To: <20120914132423.GD10160@redhat.com> References: <505065B6.50004@redhat.com> <20120914132423.GD10160@redhat.com> Message-ID: <50533810.6090600@redhat.com> On 09/14/2012 03:24 PM, Adam Tkac wrote: > On Wed, Sep 12, 2012 at 12:36:38PM +0200, Petr Spacek wrote: >> Hello, >> >> this patch adds "result" codes to error messages in persistent search code. > > Ack. Pushed to master: https://fedorahosted.org/bind-dyndb-ldap/changeset/30e3cad6267d4b57330b091001623a897478d73b Petr^2 Spacek From pspacek at redhat.com Fri Sep 14 14:00:03 2012 From: pspacek at redhat.com (Petr Spacek) Date: Fri, 14 Sep 2012 16:00:03 +0200 Subject: [Freeipa-devel] [PATCH 0059] Fix potential crash after free(uninitialized variable) In-Reply-To: <20120914133412.GA10557@redhat.com> References: <50506D0C.70706@redhat.com> <20120914133412.GA10557@redhat.com> Message-ID: <50533863.3080309@redhat.com> On 09/14/2012 03:34 PM, Adam Tkac wrote: > On Wed, Sep 12, 2012 at 01:07:56PM +0200, Petr Spacek wrote: >> Hello, >> >> This patch fixes potential crash after free(uninitialized variable) >> in persistent search code. >> >> Coverity CID 13088. > > Ack Pushed to master: https://fedorahosted.org/bind-dyndb-ldap/changeset/58fdb803e8cfd4ead174c8107ff7854a2be15b40 Petr^2 Spacek From pviktori at redhat.com Fri Sep 14 14:12:57 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Fri, 14 Sep 2012 16:12:57 +0200 Subject: [Freeipa-devel] [PATCH] 0081 Only stop the main DS instance when upgrading it In-Reply-To: <1347628337.19861.31.camel@willson.li.ssimo.org> References: <505328D1.2020806@redhat.com> <1347628337.19861.31.camel@willson.li.ssimo.org> Message-ID: <50533B69.2030905@redhat.com> On 09/14/2012 03:12 PM, Simo Sorce wrote: > On Fri, 2012-09-14 at 14:53 +0200, Petr Viktorin wrote: >> This fixes a 2.2?3.0 upgrade bug found while testing the Dogtag 10 work. >> See commit or ticket for details. >> >> https://fedorahosted.org/freeipa/ticket/3083 >> >> >> >> I also suspect that waiting for ports is not a good way to check if the >> CMS is fully initialized, but I don't know of a better way. If you know >> one, please speak up. >> > > Won;t we get back to square zero when the work to merge DS and CS > instances into one will be completed ? > > Simo. > When they're merged we'll probably need to bring down the CA while upgrading the server. Or just stop all the IPA services to be on the safe side, and of course bring them back up afterwards. Meanwhile, this patch shouldn't hurt things. -- Petr? From sbose at redhat.com Fri Sep 14 14:43:46 2012 From: sbose at redhat.com (Sumit Bose) Date: Fri, 14 Sep 2012 16:43:46 +0200 Subject: [Freeipa-devel] [PATCH] ipasam: Fixes build with samba4 rc1 Message-ID: <20120914144346.GD28400@localhost.localdomain> Hi, in samba4 rc1 there is an API change which we have to adopt in ipasam. This patch updates ipasam and unbreaks the build with samba4 rc1. bye, Sumit -------------- next part -------------- From 4e39eb306da08b29f694b9ff44ccb53865e33d92 Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Fri, 14 Sep 2012 14:14:23 +0200 Subject: [PATCH] ipasam: Fixes build with samba4 rc1 --- daemons/ipa-sam/ipa_sam.c | 20 ++++++++++---------- freeipa.spec.in | 5 ++++- 2 Dateien ge?ndert, 14 Zeilen hinzugef?gt(+), 11 Zeilen entfernt(-) diff --git a/daemons/ipa-sam/ipa_sam.c b/daemons/ipa-sam/ipa_sam.c index b3c336443d28d6850a283a373351043b2460eeaa..3d4f741c157fb624e272d800bd9e0cdf9edb9444 100644 --- a/daemons/ipa-sam/ipa_sam.c +++ b/daemons/ipa-sam/ipa_sam.c @@ -476,7 +476,7 @@ static NTSTATUS ldapsam_lookup_rids(struct pdb_methods *methods, ldap_state->ipasam_privates->base_dn, LDAP_SCOPE_SUBTREE, filter, ldap_attrs, 0, &msg); - talloc_autofree_ldapmsg(mem_ctx, msg); + smbldap_talloc_autofree_ldapmsg(mem_ctx, msg); } if (rc != LDAP_SUCCESS) @@ -546,7 +546,7 @@ static NTSTATUS ldapsam_lookup_rids(struct pdb_methods *methods, ldap_state->ipasam_privates->base_dn, LDAP_SCOPE_SUBTREE, filter, ldap_attrs, 0, &msg); - talloc_autofree_ldapmsg(mem_ctx, msg); + smbldap_talloc_autofree_ldapmsg(mem_ctx, msg); } if (rc != LDAP_SUCCESS) @@ -665,7 +665,7 @@ static bool ldapsam_sid_to_id(struct pdb_methods *methods, if (rc != LDAP_SUCCESS) { goto done; } - talloc_autofree_ldapmsg(mem_ctx, result); + smbldap_talloc_autofree_ldapmsg(mem_ctx, result); if (ldap_count_entries(priv2ld(priv), result) != 1) { DEBUG(10, ("Got %d entries, expected one\n", @@ -755,7 +755,7 @@ static bool ldapsam_uid_to_sid(struct pdb_methods *methods, uid_t uid, if (rc != LDAP_SUCCESS) { goto done; } - talloc_autofree_ldapmsg(tmp_ctx, result); + smbldap_talloc_autofree_ldapmsg(tmp_ctx, result); if (ldap_count_entries(priv2ld(priv), result) != 1) { DEBUG(3, ("ERROR: Got %d entries for uid %u, expected one\n", @@ -822,7 +822,7 @@ static bool ldapsam_gid_to_sid(struct pdb_methods *methods, gid_t gid, if (rc != LDAP_SUCCESS) { goto done; } - talloc_autofree_ldapmsg(tmp_ctx, result); + smbldap_talloc_autofree_ldapmsg(tmp_ctx, result); if (ldap_count_entries(priv2ld(priv), result) != 1) { DEBUG(3, ("ERROR: Got %d entries for gid %u, expected one\n", @@ -1482,7 +1482,7 @@ static bool search_krb_princ(struct ldapsam_privates *ldap_state, LDAP_SCOPE_SUBTREE, filter, NULL, 0, &result); if (result != NULL) { - talloc_autofree_ldapmsg(mem_ctx, result); + smbldap_talloc_autofree_ldapmsg(mem_ctx, result); } if (rc == LDAP_NO_SUCH_OBJECT) { @@ -1782,7 +1782,7 @@ static bool get_trusted_domain_int(struct ldapsam_privates *ldap_state, LDAP_SCOPE_SUBTREE, filter, NULL, 0, &result); if (result != NULL) { - talloc_autofree_ldapmsg(mem_ctx, result); + smbldap_talloc_autofree_ldapmsg(mem_ctx, result); } if (rc == LDAP_NO_SUCH_OBJECT) { @@ -2275,7 +2275,7 @@ static NTSTATUS ipasam_set_trusted_domain(struct pdb_methods *methods, &td->trust_forest_trust_info); } - talloc_autofree_ldapmod(tmp_ctx, mods); + smbldap_talloc_autofree_ldapmod(tmp_ctx, mods); trusted_dn = trusted_domain_dn(tmp_ctx, ldap_state, domain); if (trusted_dn == NULL) { @@ -2405,7 +2405,7 @@ static NTSTATUS ipasam_enum_trusted_domains(struct pdb_methods *methods, TALLOC_FREE(filter); if (result != NULL) { - talloc_autofree_ldapmsg(mem_ctx, result); + smbldap_talloc_autofree_ldapmsg(mem_ctx, result); } if (rc == LDAP_NO_SUCH_OBJECT) { @@ -2668,7 +2668,7 @@ static bool ipasam_nthash_regen(struct ldapsam_privates *ldap_state, int ret; smbldap_set_mod(&mods, LDAP_MOD_ADD, LDAP_ATTRIBUTE_NTHASH, "MagicRegen"); - talloc_autofree_ldapmod(mem_ctx, mods); + smbldap_talloc_autofree_ldapmod(mem_ctx, mods); ret = smbldap_modify(ldap_state->smbldap_state, entry_dn, mods); if (ret != LDAP_SUCCESS) { diff --git a/freeipa.spec.in b/freeipa.spec.in index dee262bbe0c5298b4a15b452214105345712e9d9..ed487f0d7124d517a2e53d4467e2d31c62cf169c 100644 --- a/freeipa.spec.in +++ b/freeipa.spec.in @@ -31,7 +31,7 @@ BuildRequires: policycoreutils >= %{POLICYCOREUTILSVER} %if 0%{?fedora} >= 16 BuildRequires: systemd-units %endif -BuildRequires: samba4-devel >= 4.0.0-134 +BuildRequires: samba4-devel >= 4.0.0-139 BuildRequires: samba4-python BuildRequires: libtalloc-devel BuildRequires: libtevent-devel @@ -752,6 +752,9 @@ fi %ghost %attr(0644,root,apache) %config(noreplace) %{_sysconfdir}/ipa/ca.crt %changelog +* Fri Sep 14 2012 Sumit Bose - 2.99.0-44 +- Updated samba4-devel dependency due to API change + * Mon Aug 20 2012 Rob Crittenden - 2.99.0-43 - Set min for 389-ds-base to 1.2.11.9-1 on F17+ to pull in warning about low nsslapd-cachememsize. -- 1.7.11.4 From simo at redhat.com Fri Sep 14 14:49:06 2012 From: simo at redhat.com (Simo Sorce) Date: Fri, 14 Sep 2012 10:49:06 -0400 Subject: [Freeipa-devel] [PATCH] ipasam: Fixes build with samba4 rc1 In-Reply-To: <20120914144346.GD28400@localhost.localdomain> References: <20120914144346.GD28400@localhost.localdomain> Message-ID: <1347634146.19861.32.camel@willson.li.ssimo.org> On Fri, 2012-09-14 at 16:43 +0200, Sumit Bose wrote: > Hi, > > in samba4 rc1 there is an API change which we have to adopt in ipasam. > This patch updates ipasam and unbreaks the build with samba4 rc1. Ack. Simo. -- Simo Sorce * Red Hat, Inc * New York From mkosek at redhat.com Fri Sep 14 14:51:11 2012 From: mkosek at redhat.com (Martin Kosek) Date: Fri, 14 Sep 2012 16:51:11 +0200 Subject: [Freeipa-devel] [PATCH] ipasam: Fixes build with samba4 rc1 In-Reply-To: <1347634146.19861.32.camel@willson.li.ssimo.org> References: <20120914144346.GD28400@localhost.localdomain> <1347634146.19861.32.camel@willson.li.ssimo.org> Message-ID: <5053445F.7020102@redhat.com> On 09/14/2012 04:49 PM, Simo Sorce wrote: > On Fri, 2012-09-14 at 16:43 +0200, Sumit Bose wrote: >> Hi, >> >> in samba4 rc1 there is an API change which we have to adopt in ipasam. >> This patch updates ipasam and unbreaks the build with samba4 rc1. > > Ack. > > Simo. > Pushed to master, ipa-3-0. Martin From pviktori at redhat.com Fri Sep 14 14:53:36 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Fri, 14 Sep 2012 16:53:36 +0200 Subject: [Freeipa-devel] [PATCH] 0081 Only stop the main DS instance when upgrading it In-Reply-To: <50533B69.2030905@redhat.com> References: <505328D1.2020806@redhat.com> <1347628337.19861.31.camel@willson.li.ssimo.org> <50533B69.2030905@redhat.com> Message-ID: <505344F0.3020901@redhat.com> On 09/14/2012 04:12 PM, Petr Viktorin wrote: > On 09/14/2012 03:12 PM, Simo Sorce wrote: >> On Fri, 2012-09-14 at 14:53 +0200, Petr Viktorin wrote: >>> This fixes a 2.2?3.0 upgrade bug found while testing the Dogtag 10 work. >>> See commit or ticket for details. >>> >>> https://fedorahosted.org/freeipa/ticket/3083 >>> >>> >>> >>> I also suspect that waiting for ports is not a good way to check if the >>> CMS is fully initialized, but I don't know of a better way. If you know >>> one, please speak up. >>> Such a better way is coming with https://fedorahosted.org/pki/ticket/314. I opened https://fedorahosted.org/freeipa/ticket/3084 so we won't forget to take advantage of it. >> Won;t we get back to square zero when the work to merge DS and CS >> instances into one will be completed ? >> >> Simo. >> > > When they're merged we'll probably need to bring down the CA while > upgrading the server. Or just stop all the IPA services to be on the > safe side, and of course bring them back up afterwards. > Meanwhile, this patch shouldn't hurt things. > -- Petr? From rcritten at redhat.com Fri Sep 14 15:13:55 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 14 Sep 2012 11:13:55 -0400 Subject: [Freeipa-devel] [PATCH] 1056 sudorule cn uniqueness In-Reply-To: <50523093.7040704@redhat.com> References: <50523093.7040704@redhat.com> Message-ID: <505349B3.10305@redhat.com> Rob Crittenden wrote: > A sudorule dn uses ipaUniqueId as the cn so we have to do a search to > ensure uniqueness. This leaves us vulnerable to a race. Configure the > uniqueness plugin to ensure no dups. > > rob Add missing attribute to the fresh install ldif. I had already fixed this in the update, missed it on fresh install. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-rcrit-1056-2-sudorule-uniqueness.patch Type: text/x-diff Size: 3287 bytes Desc: not available URL: From mkosek at redhat.com Fri Sep 14 15:37:31 2012 From: mkosek at redhat.com (Martin Kosek) Date: Fri, 14 Sep 2012 17:37:31 +0200 Subject: [Freeipa-devel] [PATCH] 0081 Only stop the main DS instance when upgrading it In-Reply-To: <505344F0.3020901@redhat.com> References: <505328D1.2020806@redhat.com> <1347628337.19861.31.camel@willson.li.ssimo.org> <50533B69.2030905@redhat.com> <505344F0.3020901@redhat.com> Message-ID: <50534F3B.6030603@redhat.com> On 09/14/2012 04:53 PM, Petr Viktorin wrote: > On 09/14/2012 04:12 PM, Petr Viktorin wrote: >> On 09/14/2012 03:12 PM, Simo Sorce wrote: >>> On Fri, 2012-09-14 at 14:53 +0200, Petr Viktorin wrote: >>>> This fixes a 2.2?3.0 upgrade bug found while testing the Dogtag 10 work. >>>> See commit or ticket for details. >>>> >>>> https://fedorahosted.org/freeipa/ticket/3083 >>>> >>>> >>>> >>>> I also suspect that waiting for ports is not a good way to check if the >>>> CMS is fully initialized, but I don't know of a better way. If you know >>>> one, please speak up. >>>> > > Such a better way is coming with https://fedorahosted.org/pki/ticket/314. I > opened https://fedorahosted.org/freeipa/ticket/3084 so we won't forget to take > advantage of it. > >>> Won;t we get back to square zero when the work to merge DS and CS >>> instances into one will be completed ? >>> >>> Simo. >>> >> >> When they're merged we'll probably need to bring down the CA while >> upgrading the server. Or just stop all the IPA services to be on the >> safe side, and of course bring them back up afterwards. >> Meanwhile, this patch shouldn't hurt things. The patch worked fine for me. It will be useful at least to the point when we use a common DS instance, as Simo suggested. Martin From rcritten at redhat.com Fri Sep 14 19:16:20 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 14 Sep 2012 15:16:20 -0400 Subject: [Freeipa-devel] [PATCH] 1050 prevent replica orphans In-Reply-To: <50531B7C.8070401@redhat.com> References: <5040F6E2.7060109@redhat.com> <5040F722.4060208@redhat.com> <5048BFAD.1070807@redhat.com> <50491414.2070504@redhat.com> <1347008704.2829.13.camel@priserak> <504E329D.6070905@redhat.com> <50531B7C.8070401@redhat.com> Message-ID: <50538284.8090205@redhat.com> Martin Kosek wrote: > On 09/10/2012 08:34 PM, Rob Crittenden wrote: >> Martin Kosek wrote: >>> On Thu, 2012-09-06 at 17:22 -0400, Rob Crittenden wrote: >>>> Martin Kosek wrote: >>>>> On 08/31/2012 07:40 PM, Rob Crittenden wrote: >>>>>> Rob Crittenden wrote: >>>>>>> It was possible use ipa-replica-manage connect/disconnect/del to end up >>>>>>> orphaning or or more IPA masters. This is an attempt to catch and >>>>>>> prevent that case. >>>>>>> >>>>>>> I tested with this topology, trying to delete B. >>>>>>> >>>>>>> A <-> B <-> C >>>>>>> >>>>>>> I got here by creating B and C from A, connecting B to C then deleting >>>>>>> the link from A to B, so it went from A -> B and A -> C to the above. >>>>>>> >>>>>>> What I do is look up the servers that the delete candidate host has >>>>>>> connections to and see if we're the last link. >>>>>>> >>>>>>> I added an escape clause if there are only two masters. >>>>>>> >>>>>>> rob >>>>>> >>>>>> Oh, this relies on my cleanruv patch 1031. >>>>>> >>>>>> rob >>>>>> >>>>> >>>>> 1) When I run ipa-replica-manage del --force to an already uninstalled host, >>>>> the new code will prevent me the deletation because it cannot connect to >>>>> it. It >>>>> also crashes with UnboundLocalError: >>>>> >>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>>> >>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing removal >>>>> Traceback (most recent call last): >>>>> File "/sbin/ipa-replica-manage", line 708, in >>>>> main() >>>>> File "/sbin/ipa-replica-manage", line 677, in main >>>>> del_master(realm, args[1], options) >>>>> File "/sbin/ipa-replica-manage", line 476, in del_master >>>>> sys.exit("Failed read master data from '%s': %s" % (delrepl.hostname, >>>>> str(e))) >>>>> UnboundLocalError: local variable 'delrepl' referenced before assignment >>>> >>>> Fixed. >>>> >>>>> >>>>> >>>>> I also hit this error when removing a winsync replica. >>>> >>>> Fixed. >>>> >>>>> >>>>> >>>>> 2) As I wrote before, I think having --force option override the user >>>>> inquiries >>>>> would benefit test automation: >>>>> >>>>> + if not ipautil.user_input("Continue to delete?", False): >>>>> + sys.exit("Aborted") >>>> >>>> Fixed. >>>> >>>>> >>>>> >>>>> 3) I don't think this code won't cover this topology: >>>>> >>>>> A - B - C - D - E >>>>> >>>>> It would allow you deleting a replica C even though it would separate A-B and >>>>> D-E. Though we may not want to cover this situation now, what you got is >>>>> definitely helping. >>>> >>>> I think you may be right. I only tested with 4 servers. With this B and >>>> D would both still have 2 agreements so wouldn't be covered by the last >>>> link test. >>> >>> Everything looks good now, so ACK. We just need to push it along with >>> CLEANALLRUV patch. >>> >>> Martin >>> >> >> Not to look a gift ACK In the mouth but here is a revised patch. I've added a >> cleanup routine to remove an orphaned master properly. If you had tried the >> mechanism I outlined in the man page it would have worked but was >> less-than-complete. This way is better, just don't try it on a live master. >> >> I also added a cleanruv abort command, in case you want to kill an existing task. >> >> rob >> > > 1) CLEANRUV stuff should be in your patch 1031 and not here (but I will comment > on the code in this mail since it is in this patch) > > > 2) In new command defitinion: > > + "abort-clean-ruv":(1, 1, "Replica ID of to abort cleaning", ""), > > I miss error message in case REPLICA_ID is not passed, this way error message > when I omit the parameter is confusing: > > # ipa-replica-manage abort-clean-ruv > Usage: ipa-replica-manage [options] > > ipa-replica-manage: error: must provide a command [force-sync | clean-ruv | > disconnect | connect | list-ruv | del | re-initialize | list | abort-clean-ruv] > > On another note, I am thinking about the new command(s). Since we now have > abort-clean-ruv command, we may want to also implement list-clean-ruv commands > in the future to see all CLEANALLRUV commands in process... Or we may enhance > list-ruv to write a flag like [CLEANALLRUV in process] for RUV's for which > CLEANALLRUV task is in process. > > > 3) Will clean-ruv - abort-clean-ruv - clean-ruv sequence work? If the aborted > CLEANALLRUV task stays in DIT, we may not be able to enter new CLEANALLRUV task > because we always use the same DN... > > Btw. did abort CLEANALLRUV command worked for you? Mine seemed to be stuck on > replicas that are down just like CLEANALLRUV command. I had then paralelly > running CLEANALLRUV and ABORT CLEANALLRUV running for the same RUV ID. Then, it > is unclear to me what this command is actually good for. > > > 4) Man page now contains non-existent command: > > --- a/install/tools/man/ipa-replica-manage.1 > +++ b/install/tools/man/ipa-replica-manage.1 > @@ -42,12 +42,18 @@ Manages the replication agreements of an IPA server. > \fBforce\-sync\fR > \- Immediately flush any data to be replicated from a server specified with > the \-\-from option > .TP > +\fBcleanup\fR > +\- Remove leftover references to a deleted master. > +.TP > > > The cleanup procedure was implemented rather as an option to the del command > than a separate command. > > > 5) My favorite - new cleanup procedure user prompt miss the --force option > useful for test automation > > + if not ipautil.user_input("Continue to clean master?", False): > + sys.exit("Cleanup aborted") > > > Otherwise the patch looks good. > > Martin > I pulled the abort-ruv stuff out. It was just easier to stuff it in here than rebasing back, but yeah, its cleaner this way. No need to check force for clean because it is already required (can't contact the deleted master, it's gone). rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-rcrit-1050-4-replicaorphan.patch Type: text/x-diff Size: 9525 bytes Desc: not available URL: From rcritten at redhat.com Fri Sep 14 19:17:35 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 14 Sep 2012 15:17:35 -0400 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <50531DDC.5050007@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> <50479491.1060200@redhat.com> <50488DD0.1080507@redhat.com> <5048BEF8.5000401@redhat.com> <5048C46A.8020207@redhat.com> <5048C754.2030708@redhat.com> <5048C9BE.1050005@redhat.com> <504912F9.1020907@redhat.com> <50531DDC.5050007@redhat.com> Message-ID: <505382CF.1030901@redhat.com> Martin Kosek wrote: > On 09/06/2012 11:17 PM, Rob Crittenden wrote: >> Martin Kosek wrote: >>> On 09/06/2012 05:55 PM, Rob Crittenden wrote: >>>> Rob Crittenden wrote: >>>>> Rob Crittenden wrote: >>>>>> Martin Kosek wrote: >>>>>>> On 09/05/2012 08:06 PM, Rob Crittenden wrote: >>>>>>>> Rob Crittenden wrote: >>>>>>>>> Martin Kosek wrote: >>>>>>>>>> On 07/05/2012 08:39 PM, Rob Crittenden wrote: >>>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: >>>>>>>>>>>>> Deleting a replica can leave a replication vector (RUV) on the >>>>>>>>>>>>> other servers. >>>>>>>>>>>>> This can confuse things if the replica is re-added, and it also >>>>>>>>>>>>> causes the >>>>>>>>>>>>> server to calculate changes against a server that may no longer >>>>>>>>>>>>> exist. >>>>>>>>>>>>> >>>>>>>>>>>>> 389-ds-base provides a new task that self-propogates itself to all >>>>>>>>>>>>> available >>>>>>>>>>>>> replicas to clean this RUV data. >>>>>>>>>>>>> >>>>>>>>>>>>> This patch will create this task at deletion time to hopefully >>>>>>>>>>>>> clean things up. >>>>>>>>>>>>> >>>>>>>>>>>>> It isn't perfect. If any replica is down or unavailable at the >>>>>>>>>>>>> time >>>>>>>>>>>>> the >>>>>>>>>>>>> cleanruv task fires, and then comes back up, the old RUV data >>>>>>>>>>>>> may be >>>>>>>>>>>>> re-propogated around. >>>>>>>>>>>>> >>>>>>>>>>>>> To make things easier in this case I've added two new commands to >>>>>>>>>>>>> ipa-replica-manage. The first lists the replication ids of all the >>>>>>>>>>>>> servers we >>>>>>>>>>>>> have a RUV for. Using this you can call clean_ruv with the >>>>>>>>>>>>> replication id of a >>>>>>>>>>>>> server that no longer exists to try the cleanallruv step again. >>>>>>>>>>>>> >>>>>>>>>>>>> This is quite dangerous though. If you run cleanruv against a >>>>>>>>>>>>> replica id that >>>>>>>>>>>>> does exist it can cause a loss of data. I believe I've put in >>>>>>>>>>>>> enough scary >>>>>>>>>>>>> warnings about this. >>>>>>>>>>>>> >>>>>>>>>>>>> rob >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Good work there, this should make cleaning RUVs much easier than >>>>>>>>>>>> with the >>>>>>>>>>>> previous version. >>>>>>>>>>>> >>>>>>>>>>>> This is what I found during review: >>>>>>>>>>>> >>>>>>>>>>>> 1) list_ruv and clean_ruv command help in man is quite lost. I >>>>>>>>>>>> think >>>>>>>>>>>> it would >>>>>>>>>>>> help if we for example have all info for commands indented. This >>>>>>>>>>>> way >>>>>>>>>>>> user could >>>>>>>>>>>> simply over-look the new commands in the man page. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2) I would rename new commands to clean-ruv and list-ruv to make >>>>>>>>>>>> them >>>>>>>>>>>> consistent with the rest of the commands (re-initialize, >>>>>>>>>>>> force-sync). >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 3) It would be nice to be able to run clean_ruv command in an >>>>>>>>>>>> unattended way >>>>>>>>>>>> (for better testing), i.e. respect --force option as we already >>>>>>>>>>>> do for >>>>>>>>>>>> ipa-replica-manage del. This fix would aid test automation in the >>>>>>>>>>>> future. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 4) (minor) The new question (and the del too) does not react too >>>>>>>>>>>> well for >>>>>>>>>>>> CTRL+D: >>>>>>>>>>>> >>>>>>>>>>>> # ipa-replica-manage clean_ruv 3 --force >>>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>>> >>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>>> Continue to clean? [no]: unexpected error: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 5) Help for clean_ruv command without a required parameter is quite >>>>>>>>>>>> confusing >>>>>>>>>>>> as it reports that command is wrong and not the parameter: >>>>>>>>>>>> >>>>>>>>>>>> # ipa-replica-manage clean_ruv >>>>>>>>>>>> Usage: ipa-replica-manage [options] >>>>>>>>>>>> >>>>>>>>>>>> ipa-replica-manage: error: must provide a command [clean_ruv | >>>>>>>>>>>> force-sync | >>>>>>>>>>>> disconnect | connect | del | re-initialize | list | list_ruv] >>>>>>>>>>>> >>>>>>>>>>>> It seems you just forgot to specify the error message in the >>>>>>>>>>>> command >>>>>>>>>>>> definition >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 6) When the remote replica is down, the clean_ruv command fails >>>>>>>>>>>> with an >>>>>>>>>>>> unexpected error: >>>>>>>>>>>> >>>>>>>>>>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 >>>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>>> >>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>>> Continue to clean? [no]: y >>>>>>>>>>>> unexpected error: {'desc': 'Operations error'} >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: >>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>> cleanAllRUV_task: failed >>>>>>>>>>>> to connect to repl agreement connection >>>>>>>>>>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, >>>>>>>>>>>> >>>>>>>>>>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> tree,cn=config), error 105 >>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>> cleanAllRUV_task: replica >>>>>>>>>>>> (cn=meTovm-055.idm.lab. >>>>>>>>>>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> tree, cn=config) has not been cleaned. You will need to rerun >>>>>>>>>>>> the >>>>>>>>>>>> CLEANALLRUV task on this replica. >>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>> cleanAllRUV_task: Task >>>>>>>>>>>> failed (1) >>>>>>>>>>>> >>>>>>>>>>>> In this case I think we should inform user that the command failed, >>>>>>>>>>>> possibly >>>>>>>>>>>> because of disconnected replicas and that they could enable the >>>>>>>>>>>> replicas and >>>>>>>>>>>> try again. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 7) (minor) "pass" is now redundant in replication.py: >>>>>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>>>>>> + # We can't make the server we're removing read-only >>>>>>>>>>>> but >>>>>>>>>>>> + # this isn't a show-stopper >>>>>>>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>>>>>>> read-only, >>>>>>>>>>>> continuing anyway") >>>>>>>>>>>> + pass >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I think this addresses everything. >>>>>>>>>>> >>>>>>>>>>> rob >>>>>>>>>> >>>>>>>>>> Thanks, almost there! I just found one more issue which needs to be >>>>>>>>>> fixed >>>>>>>>>> before we push: >>>>>>>>>> >>>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>>>>>>>> Directory Manager password: >>>>>>>>>> >>>>>>>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing >>>>>>>>>> removal >>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>>>> "Can't >>>>>>>>>> contact LDAP server"} >>>>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>>>> >>>>>>>>>> There were issues removing a connection: %d format: a number is >>>>>>>>>> required, not str >>>>>>>>>> >>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>>>> "Can't >>>>>>>>>> contact LDAP server"} >>>>>>>>>> >>>>>>>>>> This is a traceback I retrieved: >>>>>>>>>> Traceback (most recent call last): >>>>>>>>>> File "/sbin/ipa-replica-manage", line 425, in del_master >>>>>>>>>> del_link(realm, r, hostname, options.dirman_passwd, force=True) >>>>>>>>>> File "/sbin/ipa-replica-manage", line 271, in del_link >>>>>>>>>> repl1.cleanallruv(replica_id) >>>>>>>>>> File >>>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>>>>>>> line 1094, in cleanallruv >>>>>>>>>> root_logger.debug("Creating CLEANALLRUV task for replica id >>>>>>>>>> %d" % >>>>>>>>>> replicaId) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The problem here is that you don't convert replica_id to int in this >>>>>>>>>> part: >>>>>>>>>> + replica_id = None >>>>>>>>>> + if repl2: >>>>>>>>>> + replica_id = repl2._get_replica_id(repl2.conn, None) >>>>>>>>>> + else: >>>>>>>>>> + servers = get_ruv(realm, replica1, dirman_passwd) >>>>>>>>>> + for (netloc, rid) in servers: >>>>>>>>>> + if netloc.startswith(replica2): >>>>>>>>>> + replica_id = rid >>>>>>>>>> + break >>>>>>>>>> >>>>>>>>>> Martin >>>>>>>>>> >>>>>>>>> >>>>>>>>> Updated patch using new mechanism in 389-ds-base. This should more >>>>>>>>> thoroughly clean out RUV data when a replica is being deleted, and >>>>>>>>> provide for a way to delete RUV data afterwards too if necessary. >>>>>>>>> >>>>>>>>> rob >>>>>>>> >>>>>>>> Rebased patch >>>>>>>> >>>>>>>> rob >>>>>>>> >>>>>>> >>>>>>> 0) As I wrote in a review for your patch 1041, changelog entry slipped >>>>>>> elsewhere. >>>>>>> >>>>>>> 1) The following KeyboardInterrupt except class looks suspicious. I >>>>>>> know why >>>>>>> you have it there, but since it is generally a bad thing to do, some >>>>>>> comment >>>>>>> why it is needed would be useful. >>>>>>> >>>>>>> @@ -256,6 +263,17 @@ def del_link(realm, replica1, replica2, >>>>>>> dirman_passwd, >>>>>>> force=False): >>>>>>> repl1.delete_agreement(replica2) >>>>>>> repl1.delete_referral(replica2) >>>>>>> >>>>>>> + if type1 == replication.IPA_REPLICA: >>>>>>> + if repl2: >>>>>>> + ruv = repl2._get_replica_id(repl2.conn, None) >>>>>>> + else: >>>>>>> + ruv = get_ruv_by_host(realm, replica1, replica2, >>>>>>> dirman_passwd) >>>>>>> + >>>>>>> + try: >>>>>>> + repl1.cleanallruv(ruv) >>>>>>> + except KeyboardInterrupt: >>>>>>> + pass >>>>>>> + >>>>>>> >>>>>>> Maybe you just wanted to do some cleanup and then "raise" again? >>>>>> >>>>>> No, it is there because it is safe to break out of it. The task will >>>>>> continue to run. I added some verbiage. >>>>>> >>>>>>> >>>>>>> 2) This is related to 1), but when some replica is down, >>>>>>> "ipa-replica-manage >>>>>>> del" may wait indefinitely when some remote replica is down, right? >>>>>>> >>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>>> Deleting a master is irreversible. >>>>>>> To reconnect to the remote master you will need to prepare a new >>>>>>> replica file >>>>>>> and re-install. >>>>>>> Continue to delete? [no]: y >>>>>>> ipa: INFO: Setting agreement >>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>>>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> tree,cn=config >>>>>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica >>>>>>> acquired >>>>>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>>>>> Background task created to clean replication data >>>>>>> >>>>>>> ... after about a minute I hit CTRL+C >>>>>>> >>>>>>> ^CDeleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to >>>>>>> 'vm-055.idm.lab.bos.redhat.com' >>>>>>> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: NS record >>>>>>> does not >>>>>>> contain 'vm-055.idm.lab.bos.redhat.com.' >>>>>>> You may need to manually remove them from the tree >>>>>>> >>>>>>> I think it would be better to inform user that some remote replica is >>>>>>> down or >>>>>>> at least that we are waiting for the task to complete. Something like >>>>>>> that: >>>>>>> >>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>>> ... >>>>>>> Background task created to clean replication data >>>>>>> Replication data clean up may take very long time if some replica is >>>>>>> unreachable >>>>>>> Hit CTRL+C to interrupt the wait >>>>>>> ^C Clean up wait interrupted >>>>>>> .... >>>>>>> [continue with del] >>>>>> >>>>>> Yup, did this in #1. >>>>>> >>>>>>> >>>>>>> 3) (minor) When there is a cleanruv task running and you run >>>>>>> "ipa-replica-manage del", there is a unexpected error message with >>>>>>> duplicate >>>>>>> task object in LDAP: >>>>>>> >>>>>>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com --force >>>>>>> Unable to connect to replica vm-072.idm.lab.bos.redhat.com, forcing >>>>>>> removal >>>>>>> FAIL >>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>>>> contact LDAP server"} >>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>> >>>>>>> There were issues removing a connection: This entry already exists >>>>>>> <<<<<<<<< >>>>>>> >>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>>>> contact LDAP server"} >>>>>>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record >>>>>>> does not >>>>>>> contain 'vm-072.idm.lab.bos.redhat.com.' >>>>>>> You may need to manually remove them from the tree >>>>>>> >>>>>>> >>>>>>> I think it should be enough to just catch for "entry already exists" in >>>>>>> cleanallruv function, and in such case print a relevant error message >>>>>>> bail out. >>>>>>> Thus, self.conn.checkTask(dn, dowait=True) would not be called too. >>>>>> >>>>>> Good catch, fixed. >>>>>> >>>>>>> >>>>>>> >>>>>>> 4) (minor): In make_readonly function, there is a redundant "pass" >>>>>>> statement: >>>>>>> >>>>>>> + def make_readonly(self): >>>>>>> + """ >>>>>>> + Make the current replication agreement read-only. >>>>>>> + """ >>>>>>> + dn = DN(('cn', 'userRoot'), ('cn', 'ldbm database'), >>>>>>> + ('cn', 'plugins'), ('cn', 'config')) >>>>>>> + >>>>>>> + mod = [(ldap.MOD_REPLACE, 'nsslapd-readonly', 'on')] >>>>>>> + try: >>>>>>> + self.conn.modify_s(dn, mod) >>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>> + # We can't make the server we're removing read-only but >>>>>>> + # this isn't a show-stopper >>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>> read-only, >>>>>>> continuing anyway") >>>>>>> + pass <<<<<<<<<<<<<<< >>>>>> >>>>>> Yeah, this is one of my common mistakes. I put in a pass initially, then >>>>>> add logging in front of it and forget to delete the pass. Its gone now. >>>>>> >>>>>>> >>>>>>> >>>>>>> 5) In clean_ruv, I think allowing a --force option to bypass the >>>>>>> user_input >>>>>>> would be helpful (at least for test automation): >>>>>>> >>>>>>> + if not ipautil.user_input("Continue to clean?", False): >>>>>>> + sys.exit("Aborted") >>>>>> >>>>>> Yup, added. >>>>>> >>>>>> rob >>>>> >>>>> Slightly revised patch. I still had a window open with one unsaved change. >>>>> >>>>> rob >>>>> >>>> >>>> Apparently there were two unsaved changes, one of which was lost. This adds in >>>> the 'entry already exists' fix. >>>> >>>> rob >>>> >>> >>> Just one last thing (otherwise the patch is OK) - I don't think this is what we >>> want :-) >>> >>> # ipa-replica-manage clean-ruv 8 >>> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 >>> >>> Cleaning the wrong replica ID will cause that server to no >>> longer replicate so it may miss updates while the process >>> is running. It would need to be re-initialized to maintain >>> consistency. Be very careful. >>> Continue to clean? [no]: y <<<<<< >>> Aborted >>> >>> >>> Nor this exception, (your are checking for wrong exception): >>> >>> # ipa-replica-manage clean-ruv 8 >>> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 >>> >>> Cleaning the wrong replica ID will cause that server to no >>> longer replicate so it may miss updates while the process >>> is running. It would need to be re-initialized to maintain >>> consistency. Be very careful. >>> Continue to clean? [no]: >>> unexpected error: This entry already exists >>> >>> This is the exception: >>> >>> Traceback (most recent call last): >>> File "/sbin/ipa-replica-manage", line 651, in >>> main() >>> File "/sbin/ipa-replica-manage", line 648, in main >>> clean_ruv(realm, args[1], options) >>> File "/sbin/ipa-replica-manage", line 373, in clean_ruv >>> thisrepl.cleanallruv(ruv) >>> File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>> line 1136, in cleanallruv >>> self.conn.addEntry(e) >>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 503, in >>> addEntry >>> self.__handle_errors(e, arg_desc=arg_desc) >>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 321, in >>> __handle_errors >>> raise errors.DuplicateEntry() >>> ipalib.errors.DuplicateEntry: This entry already exists >>> >>> Martin >>> >> >> Fixed that and a couple of other problems. When doing a disconnect we should >> not also call clean-ruv. > > Ah, good self-catch. > >> >> I also got tired of seeing crappy error messages so I added a little convert >> utility. >> >> rob > > 1) There is CLEANALLRUV stuff included in 1050-3 and not here. There are also > some finding for this new code. > > > 2) We may want to bump Requires to higher version of 389-ds-base > (389-ds-base-1.2.11.14-1) - it contains a fix for CLEANALLRUV+winsync bug I > found earlier. > > > 3) I just discovered another suspicious behavior. When we are deleting a master > that has links also to other master(s) we delete those too. But we also > automatically run CLEANALLRUV in these cases, so we may end up in multiple > tasks being started on different masters - this does not look right. > > I think we may rather want to at first delete all links and then run > CLEANALLRUV task, just for one time. This is what I get with current code: > > # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com > Directory Manager password: > > Deleting a master is irreversible. > To reconnect to the remote master you will need to prepare a new replica file > and re-install. > Continue to delete? [no]: yes > ipa: INFO: Setting agreement > cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping > tree,cn=config schedule to 2358-2359 0 to force synch > ipa: INFO: Deleting schedule 2358-2359 0 from agreement > cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping > tree,cn=config > ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica acquired > successfully: Incremental update succeeded: start: 0: end: 0 > Background task created to clean replication data. This may take a while. > This may be safely interrupted with Ctrl+C > > ^CWait for task interrupted. It will continue to run in the background > > Deleted replication agreement from 'vm-055.idm.lab.bos.redhat.com' to > 'vm-072.idm.lab.bos.redhat.com' > ipa: INFO: Setting agreement > cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping > tree,cn=config schedule to 2358-2359 0 to force synch > ipa: INFO: Deleting schedule 2358-2359 0 from agreement > cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping > tree,cn=config > ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica acquired > successfully: Incremental update succeeded: start: 0: end: 0 > Background task created to clean replication data. This may take a while. > This may be safely interrupted with Ctrl+C > > ^CWait for task interrupted. It will continue to run in the background > > Deleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to > 'vm-072.idm.lab.bos.redhat.com' > Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record does not > contain 'vm-072.idm.lab.bos.redhat.com.' > You may need to manually remove them from the tree > > Martin > All issues addressed and I pulled in abort-clean-ruv from 1050. I added a list-clean-ruv command as well. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-rcrit-1031-9-cleanruv.patch Type: text/x-diff Size: 14380 bytes Desc: not available URL: From jfenal at gmail.com Fri Sep 14 19:36:02 2012 From: jfenal at gmail.com (=?UTF-8?B?SsOpcsO0bWUgRmVuYWw=?=) Date: Fri, 14 Sep 2012 21:36:02 +0200 Subject: [Freeipa-devel] [PATCH] 0079 Update the pot file (translation source) In-Reply-To: <50531A48.9080604@redhat.com> References: <50504AEF.9090209@redhat.com> <5052322F.2040009@redhat.com> <50531A48.9080604@redhat.com> Message-ID: 2012/9/14 Petr Viktorin > On 09/13/2012 09:21 PM, Rob Crittenden wrote: > >> Petr Viktorin wrote: >> >>> Transifex is watching our repository, so pushing this patch will update >>> the translations on the site. >>> >> > Okay, I lied. Some time ago (before I joined), Transifex changed its > "watching" mechanism from VCS pulls to URL polling. I guess IPA got > "unwatched" then. > > I pushed the pot manually. > Since we have infrequent explicit string freezes I don't think it's > necessary to configure automatic pot updates again. Thanks Petr! Actually, having the strings updated on Transifex on a regular basis makes it (IMHO) more manageable for translators to update the translations even before a string freeze. Translating a dozen strings per week is lighter than a mere 339 strings. I also don't know if pulls from Transifex or push from your side has an effect of keeping memory (in suggestions) of past or close enough strings from the past for small modifications. Another comment/request, I don't know given my zero-level Python-fu: would it be possible to break down the huge __doc__ strings in plugins into smaller parts, as a small modification would impact a smaller strings, easing maintenance instead of trying to track the one character modification in a 2000 chars text. Does Python support concatenations of __doc___ strings? Regards, J. -- J?r?me Fenal - jfenal AT gmail.com - http://fenal.org/ Paris.pm - http://paris.mongueurs.net/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Fri Sep 14 21:19:00 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 14 Sep 2012 17:19:00 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <5053350D.5010805@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> <1347329065.24731.101.camel@aleeredhat.laptop> <504F318F.3030603@redhat.com> <504F35AD.3030905@redhat.com> <1347373691.24731.125.camel@aleeredhat.laptop> <504F4CE0.9080706@redhat.com> <504F67C5.4000600@redhat.com> <504F86C0.6060907@redhat.com> <5050BB14.3050502@redhat.com> <5053350D.5010805@redhat.com> Message-ID: <50539F44.5040308@redhat.com> Petr Viktorin wrote: > On 09/12/2012 06:40 PM, Petr Viktorin wrote: >> A new Dogtag build with changed pkispawn/pkidestroy locations should be >> out later today. The attached patch should work with that build. Fresh install is failing in F-18. ki-tools-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.i686 pki-base-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.noarch pki-server-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.noarch pki-silent-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.noarch pki-symkey-9.0.21-1.fc18.x86_64 dogtag-pki-ca-theme-10.0.0-0.1.a1.20120914T0604zgit69c0684.fc18.noarch pki-selinux-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.noarch pki-ca-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.noarch pki-setup-9.0.21-1.fc18.noarch rob -------------- next part -------------- A non-text attachment was scrubbed... Name: ipaserver-install.log Type: text/x-log Size: 29293 bytes Desc: not available URL: From rcritten at redhat.com Fri Sep 14 21:57:23 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 14 Sep 2012 17:57:23 -0400 Subject: [Freeipa-devel] [PATCH] Set master_kdc and dns_lookup_kdc to true In-Reply-To: <20120907120201.GC30746@localhost.localdomain> References: <20120907120201.GC30746@localhost.localdomain> Message-ID: <5053A843.7070005@redhat.com> Sumit Bose wrote: > Hi, > > those two patches should fix > https://fedorahosted.org/freeipa/ticket/2515 . The first makes the > needed change for fresh installations. The second adds the changes > during ipa-adtrust-install if needed. I prefer to do the changes here > instead of during updates, because during updates it is not easy to see > that the Kerberos configuration was changes. > I guess it is good form to update the RHEL 4 client installer but will anyone test it? Is master_kdc supported in the MIT kfw version (krb5.ini)? This suffers from the problem Simo envisioned with ticket 931. If the /etc/hosts entry is removed then DNS will not start. We add an entry during installation, so this may be less of an issue. rob From jhrozek at redhat.com Sat Sep 15 09:55:25 2012 From: jhrozek at redhat.com (Jakub Hrozek) Date: Sat, 15 Sep 2012 11:55:25 +0200 Subject: [Freeipa-devel] [PATCH] Set master_kdc and dns_lookup_kdc to true In-Reply-To: <5053A843.7070005@redhat.com> References: <20120907120201.GC30746@localhost.localdomain> <5053A843.7070005@redhat.com> Message-ID: <20120915095525.GD26821@hendrix.redhat.com> On Fri, Sep 14, 2012 at 05:57:23PM -0400, Rob Crittenden wrote: > Sumit Bose wrote: > >Hi, > > > >those two patches should fix > >https://fedorahosted.org/freeipa/ticket/2515 . The first makes the > >needed change for fresh installations. The second adds the changes > >during ipa-adtrust-install if needed. I prefer to do the changes here > >instead of during updates, because during updates it is not easy to see > >that the Kerberos configuration was changes. > > > > I guess it is good form to update the RHEL 4 client installer but > will anyone test it? Can we just remove it? I think that RHEL4 has been EOLd already: https://rhn.redhat.com/errata/RHSA-2012-0073.html From sbose at redhat.com Sat Sep 15 20:02:10 2012 From: sbose at redhat.com (Sumit Bose) Date: Sat, 15 Sep 2012 22:02:10 +0200 Subject: [Freeipa-devel] [PATCH] Set master_kdc and dns_lookup_kdc to true In-Reply-To: <5053A843.7070005@redhat.com> References: <20120907120201.GC30746@localhost.localdomain> <5053A843.7070005@redhat.com> Message-ID: <20120915200210.GE28400@localhost.localdomain> On Fri, Sep 14, 2012 at 05:57:23PM -0400, Rob Crittenden wrote: > Sumit Bose wrote: > >Hi, > > > >those two patches should fix > >https://fedorahosted.org/freeipa/ticket/2515 . The first makes the > >needed change for fresh installations. The second adds the changes > >during ipa-adtrust-install if needed. I prefer to do the changes here > >instead of during updates, because during updates it is not easy to see > >that the Kerberos configuration was changes. > > > > I guess it is good form to update the RHEL 4 client installer but > will anyone test it? I think it would be confusion if the RHEL4 client installer has different information than the default one. > > Is master_kdc supported in the MIT kfw version (krb5.ini)? For me it looks that the parse is build from the same sources. > > This suffers from the problem Simo envisioned with ticket 931. If > the /etc/hosts entry is removed then DNS will not start. We add an > entry during installation, so this may be less of an issue. If the /etc/hosts entry is removed DNS will not start in either case. I think the solution to #931 is setting the master_kdc option. You can easily reproduce startup problems if you set 'dns_lookup_kdc = true', stop sssd and try to restart named. This will run into a timeout and bind will not start. The reason is that besides a KDC the Kerberos client libraries also try to look up the master KDC (but it seems to be ok if the lookup finally fails). If sssd is running the locator plugin will return the current KDC as master. If it is not running, as in the test described above, /etc/krb5.conf is used next. If it does not have a master_kdc entry and 'dns_lookup_kdc = false' there is no other source for the master KDC and the client libraries continue with normal processing. If master_kdc is not set but 'dns_lookup_kdc = true' then a DNS lookup is tried, which will run into a timeout since the DNS server is not started yet. But if master_kdc is set in krb5.conf the client libraries will just use this value and will not try any DNS lookup, independently of the setting of dns_lookup_kdc. As a side note. Since we run named as user named I wonder if it would be possible to use SASL EXTERNAL auth instead of GSSAPI to bind to the LDAP server. If this would work safe and secure there would be no dependencies to the KDC during the startup of bind? bye, Sumit > > rob > From simo at redhat.com Sat Sep 15 22:14:56 2012 From: simo at redhat.com (Simo Sorce) Date: Sat, 15 Sep 2012 18:14:56 -0400 Subject: [Freeipa-devel] [PATCH] Set master_kdc and dns_lookup_kdc to true In-Reply-To: <20120915200210.GE28400@localhost.localdomain> References: <20120907120201.GC30746@localhost.localdomain> <5053A843.7070005@redhat.com> <20120915200210.GE28400@localhost.localdomain> Message-ID: <1347747296.19861.78.camel@willson.li.ssimo.org> On Sat, 2012-09-15 at 22:02 +0200, Sumit Bose wrote: > On Fri, Sep 14, 2012 at 05:57:23PM -0400, Rob Crittenden wrote: > > Sumit Bose wrote: > > >Hi, > > > > > >those two patches should fix > > >https://fedorahosted.org/freeipa/ticket/2515 . The first makes the > > >needed change for fresh installations. The second adds the changes > > >during ipa-adtrust-install if needed. I prefer to do the changes here > > >instead of during updates, because during updates it is not easy to see > > >that the Kerberos configuration was changes. > > > > > > > I guess it is good form to update the RHEL 4 client installer but > > will anyone test it? > > I think it would be confusion if the RHEL4 client installer has > different information than the default one. > > > > > Is master_kdc supported in the MIT kfw version (krb5.ini)? > > For me it looks that the parse is build from the same sources. > > > > > This suffers from the problem Simo envisioned with ticket 931. If > > the /etc/hosts entry is removed then DNS will not start. We add an > > entry during installation, so this may be less of an issue. > > If the /etc/hosts entry is removed DNS will not start in either case. > > I think the solution to #931 is setting the master_kdc option. You can > easily reproduce startup problems if you set 'dns_lookup_kdc = true', > stop sssd and try to restart named. This will run into a timeout and > bind will not start. The reason is that besides a KDC the Kerberos > client libraries also try to look up the master KDC (but it seems to be > ok if the lookup finally fails). If sssd is running the locator plugin > will return the current KDC as master. If it is not running, as in the > test described above, /etc/krb5.conf is used next. If it does not have a > master_kdc entry and 'dns_lookup_kdc = false' there is no other source > for the master KDC and the client libraries continue with normal > processing. If master_kdc is not set but 'dns_lookup_kdc = true' then a > DNS lookup is tried, which will run into a timeout since the DNS server > is not started yet. But if master_kdc is set in krb5.conf the client > libraries will just use this value and will not try any DNS lookup, > independently of the setting of dns_lookup_kdc. > > As a side note. Since we run named as user named I wonder if it would be > possible to use SASL EXTERNAL auth instead of GSSAPI to bind to the LDAP > server. If this would work safe and secure there would be no > dependencies to the KDC during the startup of bind? The reason why we use gssapi is so that all operations performed by bind happen as the DNS/fqdn user, and we can use ACIs targeted at the bind process. In order to use SASL EXTERNAL we would need the bind process to change euid to an unprivileged user that we then need to map to some specific user. In general krb5kdc should always start before named, and should not depend on DNS resolution. If krb5kdc is started first bind should have no issues. The only proble is if gssapi libraries try to use DNS resolution, but we should have that solved by using the krb locator plugin. Simo. -- Simo Sorce * Red Hat, Inc * New York From william at firstyear.id.au Mon Sep 17 01:34:52 2012 From: william at firstyear.id.au (William Brown) Date: Mon, 17 Sep 2012 11:04:52 +0930 Subject: [Freeipa-devel] Automatic keytab upgrade Message-ID: <50567E3C.5080606@firstyear.id.au> Hi, In freeipa, we export a keytab for a service. Say we upgrade our freeipa install, and the newer version of MIT kerberos supports a stronger encryption type on the KDC. Does freeipa automatically refresh the keytabs of hosts / services with a new keytab that also contains these stronger encryption types? Does this matter if it does / doesn't happen? -- Sincerely, William Brown pgp.mit.edu http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=0x3C0AC6DAB2F928A2 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 937 bytes Desc: OpenPGP digital signature URL: From pspacek at redhat.com Mon Sep 17 07:06:09 2012 From: pspacek at redhat.com (Petr Spacek) Date: Mon, 17 Sep 2012 09:06:09 +0200 Subject: [Freeipa-devel] IPA server resolv.conf (was: [PATCH] Set master_kdc and dns_lookup_kdc to true) In-Reply-To: <1347747296.19861.78.camel@willson.li.ssimo.org> References: <1347747296.19861.78.camel@willson.li.ssimo.org> Message-ID: <5056CBE1.3000100@redhat.com> Discussion about patch "Set master_kdc and dns_lookup_kdc to true)" reminds one related problem: Our server installer puts line "nameserver 127.0.0.1" to /etc/resolv.conf, but this file should contain all (or three nearest) DNS servers in IPA domain. As a result, IPA server will work even after local named crash (which is not so rare as I want :-(). New ticket: https://fedorahosted.org/freeipa/ticket/3085 Martin, what do you think? How we can update resolv.conf to reflect replica addition/deletion? Should it be done manually? E.g. ipa-replica-install script can print "don't forget to add this server to /etc/resolv.conf on other servers"? Petr^2 Spacek -------- Original Message -------- Subject: Re: [Freeipa-devel] [PATCH] Set master_kdc and dns_lookup_kdc to true Date: Sat, 15 Sep 2012 18:14:56 -0400 From: Simo Sorce Organization: Red Hat, Inc. To: Sumit Bose CC: freeipa-devel On Sat, 2012-09-15 at 22:02 +0200, Sumit Bose wrote: > On Fri, Sep 14, 2012 at 05:57:23PM -0400, Rob Crittenden wrote: > > Sumit Bose wrote: > > >Hi, > > > > > >those two patches should fix > > >https://fedorahosted.org/freeipa/ticket/2515 . The first makes the > > >needed change for fresh installations. The second adds the changes > > >during ipa-adtrust-install if needed. I prefer to do the changes here > > >instead of during updates, because during updates it is not easy to see > > >that the Kerberos configuration was changes. > > > > > > > I guess it is good form to update the RHEL 4 client installer but > > will anyone test it? > > I think it would be confusion if the RHEL4 client installer has > different information than the default one. > > > > > Is master_kdc supported in the MIT kfw version (krb5.ini)? > > For me it looks that the parse is build from the same sources. > > > > > This suffers from the problem Simo envisioned with ticket 931. If > > the /etc/hosts entry is removed then DNS will not start. We add an > > entry during installation, so this may be less of an issue. > > If the /etc/hosts entry is removed DNS will not start in either case. > > I think the solution to #931 is setting the master_kdc option. You can > easily reproduce startup problems if you set 'dns_lookup_kdc = true', > stop sssd and try to restart named. This will run into a timeout and > bind will not start. The reason is that besides a KDC the Kerberos > client libraries also try to look up the master KDC (but it seems to be > ok if the lookup finally fails). If sssd is running the locator plugin > will return the current KDC as master. If it is not running, as in the > test described above, /etc/krb5.conf is used next. If it does not have a > master_kdc entry and 'dns_lookup_kdc = false' there is no other source > for the master KDC and the client libraries continue with normal > processing. If master_kdc is not set but 'dns_lookup_kdc = true' then a > DNS lookup is tried, which will run into a timeout since the DNS server > is not started yet. But if master_kdc is set in krb5.conf the client > libraries will just use this value and will not try any DNS lookup, > independently of the setting of dns_lookup_kdc. > > As a side note. Since we run named as user named I wonder if it would be > possible to use SASL EXTERNAL auth instead of GSSAPI to bind to the LDAP > server. If this would work safe and secure there would be no > dependencies to the KDC during the startup of bind? The reason why we use gssapi is so that all operations performed by bind happen as the DNS/fqdn user, and we can use ACIs targeted at the bind process. In order to use SASL EXTERNAL we would need the bind process to change euid to an unprivileged user that we then need to map to some specific user. In general krb5kdc should always start before named, and should not depend on DNS resolution. If krb5kdc is started first bind should have no issues. The only proble is if gssapi libraries try to use DNS resolution, but we should have that solved by using the krb locator plugin. Simo. -- Simo Sorce * Red Hat, Inc * New York _______________________________________________ Freeipa-devel mailing list Freeipa-devel at redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel From mkosek at redhat.com Mon Sep 17 07:15:55 2012 From: mkosek at redhat.com (Martin Kosek) Date: Mon, 17 Sep 2012 09:15:55 +0200 Subject: [Freeipa-devel] IPA server resolv.conf In-Reply-To: <5056CBE1.3000100@redhat.com> References: <1347747296.19861.78.camel@willson.li.ssimo.org> <5056CBE1.3000100@redhat.com> Message-ID: <5056CE2B.4060407@redhat.com> On 09/17/2012 09:06 AM, Petr Spacek wrote: > Discussion about patch "Set master_kdc and dns_lookup_kdc to true)" reminds one > related problem: > > Our server installer puts line "nameserver 127.0.0.1" to /etc/resolv.conf, but > this file should contain all (or three nearest) DNS servers in IPA domain. > > As a result, IPA server will work even after local named crash (which is not so > rare as I want :-(). > > New ticket: > https://fedorahosted.org/freeipa/ticket/3085 > > Martin, what do you think? > > How we can update resolv.conf to reflect replica addition/deletion? > > Should it be done manually? E.g. ipa-replica-install script can print "don't > forget to add this server to /etc/resolv.conf on other servers"? > > Petr^2 Spacek > It would not be difficult to pull a list of IPA masters with DNS support during ipa-{server,replica}-install and write more IPs to the resolv.conf. But I think there may be an issue when somebody willingly stop a remote replica or uninstall it. He would also need to remove it's IP from all resolv.confs in all replicas... Btw. why would IPA server fail when a local named crashes? A record in /etc/hosts we always add should still enable local IPA services to work or do I miss something? Martin From pviktori at redhat.com Mon Sep 17 08:18:33 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Mon, 17 Sep 2012 10:18:33 +0200 Subject: [Freeipa-devel] [PATCH] 0079 Update the pot file (translation source) In-Reply-To: References: <50504AEF.9090209@redhat.com> <5052322F.2040009@redhat.com> <50531A48.9080604@redhat.com> Message-ID: <5056DCD9.9020805@redhat.com> On 09/14/2012 09:36 PM, J?r?me Fenal wrote: > 2012/9/14 Petr Viktorin > > I pushed the pot manually. > Since we have infrequent explicit string freezes I don't think it's > necessary to configure automatic pot updates again. > > > Thanks Petr! > > Actually, having the strings updated on Transifex on a regular basis > makes it (IMHO) more manageable for translators to update the > translations even before a string freeze. Translating a dozen strings > per week is lighter than a mere 339 strings. A possible problem with this approach is that the translators would see and translate messages that don't make it into the final version. Do you think a more even workload would be worth the occasional extra work? I would like to change our i18n workflow/infrastructure. I was planning to (re-)start discussing this after the 3.0 release rush is done. It should be possible to do what you suggest. > I also don't know if pulls from Transifex or push from your side has an > effect of keeping memory (in suggestions) of past or close enough > strings from the past for small modifications. Sadly, I don't know much about Transifex itself. Perhaps ask the team there, and request the feature if it's missing. > Another comment/request, I don't know given my zero-level Python-fu: > would it be possible to break down the huge __doc__ strings in plugins > into smaller parts, as a small modification would impact a smaller > strings, easing maintenance instead of trying to track the one character > modification in a 2000 chars text. > > Does Python support concatenations of __doc___ strings? That is be possible on the Python side. I'm not sure how Transifex (and other translation tools) would cope with text split between several messages -- sorting and filtering the messages could take things out of context. -- Petr? From mkosek at redhat.com Mon Sep 17 09:03:16 2012 From: mkosek at redhat.com (Martin Kosek) Date: Mon, 17 Sep 2012 11:03:16 +0200 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <505382CF.1030901@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> <50479491.1060200@redhat.com> <50488DD0.1080507@redhat.com> <5048BEF8.5000401@redhat.com> <5048C46A.8020207@redhat.com> <5048C754.2030708@redhat.com> <5048C9BE.1050005@redhat.com> <504912F9.1020907@redhat.com> <50531DDC.5050007@redhat.com> <505382CF.1030901@redhat.com> Message-ID: <5056E754.1080201@redhat.com> On 09/14/2012 09:17 PM, Rob Crittenden wrote: > Martin Kosek wrote: >> On 09/06/2012 11:17 PM, Rob Crittenden wrote: >>> Martin Kosek wrote: >>>> On 09/06/2012 05:55 PM, Rob Crittenden wrote: >>>>> Rob Crittenden wrote: >>>>>> Rob Crittenden wrote: >>>>>>> Martin Kosek wrote: >>>>>>>> On 09/05/2012 08:06 PM, Rob Crittenden wrote: >>>>>>>>> Rob Crittenden wrote: >>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>> On 07/05/2012 08:39 PM, Rob Crittenden wrote: >>>>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: >>>>>>>>>>>>>> Deleting a replica can leave a replication vector (RUV) on the >>>>>>>>>>>>>> other servers. >>>>>>>>>>>>>> This can confuse things if the replica is re-added, and it also >>>>>>>>>>>>>> causes the >>>>>>>>>>>>>> server to calculate changes against a server that may no longer >>>>>>>>>>>>>> exist. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 389-ds-base provides a new task that self-propogates itself to all >>>>>>>>>>>>>> available >>>>>>>>>>>>>> replicas to clean this RUV data. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This patch will create this task at deletion time to hopefully >>>>>>>>>>>>>> clean things up. >>>>>>>>>>>>>> >>>>>>>>>>>>>> It isn't perfect. If any replica is down or unavailable at the >>>>>>>>>>>>>> time >>>>>>>>>>>>>> the >>>>>>>>>>>>>> cleanruv task fires, and then comes back up, the old RUV data >>>>>>>>>>>>>> may be >>>>>>>>>>>>>> re-propogated around. >>>>>>>>>>>>>> >>>>>>>>>>>>>> To make things easier in this case I've added two new commands to >>>>>>>>>>>>>> ipa-replica-manage. The first lists the replication ids of all the >>>>>>>>>>>>>> servers we >>>>>>>>>>>>>> have a RUV for. Using this you can call clean_ruv with the >>>>>>>>>>>>>> replication id of a >>>>>>>>>>>>>> server that no longer exists to try the cleanallruv step again. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is quite dangerous though. If you run cleanruv against a >>>>>>>>>>>>>> replica id that >>>>>>>>>>>>>> does exist it can cause a loss of data. I believe I've put in >>>>>>>>>>>>>> enough scary >>>>>>>>>>>>>> warnings about this. >>>>>>>>>>>>>> >>>>>>>>>>>>>> rob >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Good work there, this should make cleaning RUVs much easier than >>>>>>>>>>>>> with the >>>>>>>>>>>>> previous version. >>>>>>>>>>>>> >>>>>>>>>>>>> This is what I found during review: >>>>>>>>>>>>> >>>>>>>>>>>>> 1) list_ruv and clean_ruv command help in man is quite lost. I >>>>>>>>>>>>> think >>>>>>>>>>>>> it would >>>>>>>>>>>>> help if we for example have all info for commands indented. This >>>>>>>>>>>>> way >>>>>>>>>>>>> user could >>>>>>>>>>>>> simply over-look the new commands in the man page. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 2) I would rename new commands to clean-ruv and list-ruv to make >>>>>>>>>>>>> them >>>>>>>>>>>>> consistent with the rest of the commands (re-initialize, >>>>>>>>>>>>> force-sync). >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 3) It would be nice to be able to run clean_ruv command in an >>>>>>>>>>>>> unattended way >>>>>>>>>>>>> (for better testing), i.e. respect --force option as we already >>>>>>>>>>>>> do for >>>>>>>>>>>>> ipa-replica-manage del. This fix would aid test automation in the >>>>>>>>>>>>> future. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 4) (minor) The new question (and the del too) does not react too >>>>>>>>>>>>> well for >>>>>>>>>>>>> CTRL+D: >>>>>>>>>>>>> >>>>>>>>>>>>> # ipa-replica-manage clean_ruv 3 --force >>>>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>>>> >>>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>>>> Continue to clean? [no]: unexpected error: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 5) Help for clean_ruv command without a required parameter is quite >>>>>>>>>>>>> confusing >>>>>>>>>>>>> as it reports that command is wrong and not the parameter: >>>>>>>>>>>>> >>>>>>>>>>>>> # ipa-replica-manage clean_ruv >>>>>>>>>>>>> Usage: ipa-replica-manage [options] >>>>>>>>>>>>> >>>>>>>>>>>>> ipa-replica-manage: error: must provide a command [clean_ruv | >>>>>>>>>>>>> force-sync | >>>>>>>>>>>>> disconnect | connect | del | re-initialize | list | list_ruv] >>>>>>>>>>>>> >>>>>>>>>>>>> It seems you just forgot to specify the error message in the >>>>>>>>>>>>> command >>>>>>>>>>>>> definition >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 6) When the remote replica is down, the clean_ruv command fails >>>>>>>>>>>>> with an >>>>>>>>>>>>> unexpected error: >>>>>>>>>>>>> >>>>>>>>>>>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 >>>>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>>>> >>>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>>>> Continue to clean? [no]: y >>>>>>>>>>>>> unexpected error: {'desc': 'Operations error'} >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: >>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>> cleanAllRUV_task: failed >>>>>>>>>>>>> to connect to repl agreement connection >>>>>>>>>>>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, >>>>>>>>>>>>> >>>>>>>>>>>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> tree,cn=config), error 105 >>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>> cleanAllRUV_task: replica >>>>>>>>>>>>> (cn=meTovm-055.idm.lab. >>>>>>>>>>>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> tree, cn=config) has not been cleaned. You will need to rerun >>>>>>>>>>>>> the >>>>>>>>>>>>> CLEANALLRUV task on this replica. >>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>> cleanAllRUV_task: Task >>>>>>>>>>>>> failed (1) >>>>>>>>>>>>> >>>>>>>>>>>>> In this case I think we should inform user that the command failed, >>>>>>>>>>>>> possibly >>>>>>>>>>>>> because of disconnected replicas and that they could enable the >>>>>>>>>>>>> replicas and >>>>>>>>>>>>> try again. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 7) (minor) "pass" is now redundant in replication.py: >>>>>>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>>>>>>> + # We can't make the server we're removing read-only >>>>>>>>>>>>> but >>>>>>>>>>>>> + # this isn't a show-stopper >>>>>>>>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>>>>>>>> read-only, >>>>>>>>>>>>> continuing anyway") >>>>>>>>>>>>> + pass >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I think this addresses everything. >>>>>>>>>>>> >>>>>>>>>>>> rob >>>>>>>>>>> >>>>>>>>>>> Thanks, almost there! I just found one more issue which needs to be >>>>>>>>>>> fixed >>>>>>>>>>> before we push: >>>>>>>>>>> >>>>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>>>>>>>>> Directory Manager password: >>>>>>>>>>> >>>>>>>>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing >>>>>>>>>>> removal >>>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>>>>> "Can't >>>>>>>>>>> contact LDAP server"} >>>>>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>>>>> >>>>>>>>>>> There were issues removing a connection: %d format: a number is >>>>>>>>>>> required, not str >>>>>>>>>>> >>>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>>>>> "Can't >>>>>>>>>>> contact LDAP server"} >>>>>>>>>>> >>>>>>>>>>> This is a traceback I retrieved: >>>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>> File "/sbin/ipa-replica-manage", line 425, in del_master >>>>>>>>>>> del_link(realm, r, hostname, options.dirman_passwd, force=True) >>>>>>>>>>> File "/sbin/ipa-replica-manage", line 271, in del_link >>>>>>>>>>> repl1.cleanallruv(replica_id) >>>>>>>>>>> File >>>>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>>>>>>>> line 1094, in cleanallruv >>>>>>>>>>> root_logger.debug("Creating CLEANALLRUV task for replica id >>>>>>>>>>> %d" % >>>>>>>>>>> replicaId) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> The problem here is that you don't convert replica_id to int in this >>>>>>>>>>> part: >>>>>>>>>>> + replica_id = None >>>>>>>>>>> + if repl2: >>>>>>>>>>> + replica_id = repl2._get_replica_id(repl2.conn, None) >>>>>>>>>>> + else: >>>>>>>>>>> + servers = get_ruv(realm, replica1, dirman_passwd) >>>>>>>>>>> + for (netloc, rid) in servers: >>>>>>>>>>> + if netloc.startswith(replica2): >>>>>>>>>>> + replica_id = rid >>>>>>>>>>> + break >>>>>>>>>>> >>>>>>>>>>> Martin >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Updated patch using new mechanism in 389-ds-base. This should more >>>>>>>>>> thoroughly clean out RUV data when a replica is being deleted, and >>>>>>>>>> provide for a way to delete RUV data afterwards too if necessary. >>>>>>>>>> >>>>>>>>>> rob >>>>>>>>> >>>>>>>>> Rebased patch >>>>>>>>> >>>>>>>>> rob >>>>>>>>> >>>>>>>> >>>>>>>> 0) As I wrote in a review for your patch 1041, changelog entry slipped >>>>>>>> elsewhere. >>>>>>>> >>>>>>>> 1) The following KeyboardInterrupt except class looks suspicious. I >>>>>>>> know why >>>>>>>> you have it there, but since it is generally a bad thing to do, some >>>>>>>> comment >>>>>>>> why it is needed would be useful. >>>>>>>> >>>>>>>> @@ -256,6 +263,17 @@ def del_link(realm, replica1, replica2, >>>>>>>> dirman_passwd, >>>>>>>> force=False): >>>>>>>> repl1.delete_agreement(replica2) >>>>>>>> repl1.delete_referral(replica2) >>>>>>>> >>>>>>>> + if type1 == replication.IPA_REPLICA: >>>>>>>> + if repl2: >>>>>>>> + ruv = repl2._get_replica_id(repl2.conn, None) >>>>>>>> + else: >>>>>>>> + ruv = get_ruv_by_host(realm, replica1, replica2, >>>>>>>> dirman_passwd) >>>>>>>> + >>>>>>>> + try: >>>>>>>> + repl1.cleanallruv(ruv) >>>>>>>> + except KeyboardInterrupt: >>>>>>>> + pass >>>>>>>> + >>>>>>>> >>>>>>>> Maybe you just wanted to do some cleanup and then "raise" again? >>>>>>> >>>>>>> No, it is there because it is safe to break out of it. The task will >>>>>>> continue to run. I added some verbiage. >>>>>>> >>>>>>>> >>>>>>>> 2) This is related to 1), but when some replica is down, >>>>>>>> "ipa-replica-manage >>>>>>>> del" may wait indefinitely when some remote replica is down, right? >>>>>>>> >>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>>>> Deleting a master is irreversible. >>>>>>>> To reconnect to the remote master you will need to prepare a new >>>>>>>> replica file >>>>>>>> and re-install. >>>>>>>> Continue to delete? [no]: y >>>>>>>> ipa: INFO: Setting agreement >>>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>>>>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> tree,cn=config >>>>>>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica >>>>>>>> acquired >>>>>>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>>>>>> Background task created to clean replication data >>>>>>>> >>>>>>>> ... after about a minute I hit CTRL+C >>>>>>>> >>>>>>>> ^CDeleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to >>>>>>>> 'vm-055.idm.lab.bos.redhat.com' >>>>>>>> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: NS record >>>>>>>> does not >>>>>>>> contain 'vm-055.idm.lab.bos.redhat.com.' >>>>>>>> You may need to manually remove them from the tree >>>>>>>> >>>>>>>> I think it would be better to inform user that some remote replica is >>>>>>>> down or >>>>>>>> at least that we are waiting for the task to complete. Something like >>>>>>>> that: >>>>>>>> >>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>>>> ... >>>>>>>> Background task created to clean replication data >>>>>>>> Replication data clean up may take very long time if some replica is >>>>>>>> unreachable >>>>>>>> Hit CTRL+C to interrupt the wait >>>>>>>> ^C Clean up wait interrupted >>>>>>>> .... >>>>>>>> [continue with del] >>>>>>> >>>>>>> Yup, did this in #1. >>>>>>> >>>>>>>> >>>>>>>> 3) (minor) When there is a cleanruv task running and you run >>>>>>>> "ipa-replica-manage del", there is a unexpected error message with >>>>>>>> duplicate >>>>>>>> task object in LDAP: >>>>>>>> >>>>>>>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com --force >>>>>>>> Unable to connect to replica vm-072.idm.lab.bos.redhat.com, forcing >>>>>>>> removal >>>>>>>> FAIL >>>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>>>>> contact LDAP server"} >>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>> >>>>>>>> There were issues removing a connection: This entry already exists >>>>>>>> <<<<<<<<< >>>>>>>> >>>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>>>>> contact LDAP server"} >>>>>>>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record >>>>>>>> does not >>>>>>>> contain 'vm-072.idm.lab.bos.redhat.com.' >>>>>>>> You may need to manually remove them from the tree >>>>>>>> >>>>>>>> >>>>>>>> I think it should be enough to just catch for "entry already exists" in >>>>>>>> cleanallruv function, and in such case print a relevant error message >>>>>>>> bail out. >>>>>>>> Thus, self.conn.checkTask(dn, dowait=True) would not be called too. >>>>>>> >>>>>>> Good catch, fixed. >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> 4) (minor): In make_readonly function, there is a redundant "pass" >>>>>>>> statement: >>>>>>>> >>>>>>>> + def make_readonly(self): >>>>>>>> + """ >>>>>>>> + Make the current replication agreement read-only. >>>>>>>> + """ >>>>>>>> + dn = DN(('cn', 'userRoot'), ('cn', 'ldbm database'), >>>>>>>> + ('cn', 'plugins'), ('cn', 'config')) >>>>>>>> + >>>>>>>> + mod = [(ldap.MOD_REPLACE, 'nsslapd-readonly', 'on')] >>>>>>>> + try: >>>>>>>> + self.conn.modify_s(dn, mod) >>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>> + # We can't make the server we're removing read-only but >>>>>>>> + # this isn't a show-stopper >>>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>>> read-only, >>>>>>>> continuing anyway") >>>>>>>> + pass <<<<<<<<<<<<<<< >>>>>>> >>>>>>> Yeah, this is one of my common mistakes. I put in a pass initially, then >>>>>>> add logging in front of it and forget to delete the pass. Its gone now. >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> 5) In clean_ruv, I think allowing a --force option to bypass the >>>>>>>> user_input >>>>>>>> would be helpful (at least for test automation): >>>>>>>> >>>>>>>> + if not ipautil.user_input("Continue to clean?", False): >>>>>>>> + sys.exit("Aborted") >>>>>>> >>>>>>> Yup, added. >>>>>>> >>>>>>> rob >>>>>> >>>>>> Slightly revised patch. I still had a window open with one unsaved change. >>>>>> >>>>>> rob >>>>>> >>>>> >>>>> Apparently there were two unsaved changes, one of which was lost. This >>>>> adds in >>>>> the 'entry already exists' fix. >>>>> >>>>> rob >>>>> >>>> >>>> Just one last thing (otherwise the patch is OK) - I don't think this is >>>> what we >>>> want :-) >>>> >>>> # ipa-replica-manage clean-ruv 8 >>>> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 >>>> >>>> Cleaning the wrong replica ID will cause that server to no >>>> longer replicate so it may miss updates while the process >>>> is running. It would need to be re-initialized to maintain >>>> consistency. Be very careful. >>>> Continue to clean? [no]: y <<<<<< >>>> Aborted >>>> >>>> >>>> Nor this exception, (your are checking for wrong exception): >>>> >>>> # ipa-replica-manage clean-ruv 8 >>>> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 >>>> >>>> Cleaning the wrong replica ID will cause that server to no >>>> longer replicate so it may miss updates while the process >>>> is running. It would need to be re-initialized to maintain >>>> consistency. Be very careful. >>>> Continue to clean? [no]: >>>> unexpected error: This entry already exists >>>> >>>> This is the exception: >>>> >>>> Traceback (most recent call last): >>>> File "/sbin/ipa-replica-manage", line 651, in >>>> main() >>>> File "/sbin/ipa-replica-manage", line 648, in main >>>> clean_ruv(realm, args[1], options) >>>> File "/sbin/ipa-replica-manage", line 373, in clean_ruv >>>> thisrepl.cleanallruv(ruv) >>>> File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>> line 1136, in cleanallruv >>>> self.conn.addEntry(e) >>>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 503, in >>>> addEntry >>>> self.__handle_errors(e, arg_desc=arg_desc) >>>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 321, in >>>> __handle_errors >>>> raise errors.DuplicateEntry() >>>> ipalib.errors.DuplicateEntry: This entry already exists >>>> >>>> Martin >>>> >>> >>> Fixed that and a couple of other problems. When doing a disconnect we should >>> not also call clean-ruv. >> >> Ah, good self-catch. >> >>> >>> I also got tired of seeing crappy error messages so I added a little convert >>> utility. >>> >>> rob >> >> 1) There is CLEANALLRUV stuff included in 1050-3 and not here. There are also >> some finding for this new code. >> >> >> 2) We may want to bump Requires to higher version of 389-ds-base >> (389-ds-base-1.2.11.14-1) - it contains a fix for CLEANALLRUV+winsync bug I >> found earlier. >> >> >> 3) I just discovered another suspicious behavior. When we are deleting a master >> that has links also to other master(s) we delete those too. But we also >> automatically run CLEANALLRUV in these cases, so we may end up in multiple >> tasks being started on different masters - this does not look right. >> >> I think we may rather want to at first delete all links and then run >> CLEANALLRUV task, just for one time. This is what I get with current code: >> >> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com >> Directory Manager password: >> >> Deleting a master is irreversible. >> To reconnect to the remote master you will need to prepare a new replica file >> and re-install. >> Continue to delete? [no]: yes >> ipa: INFO: Setting agreement >> cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >> >> tree,cn=config schedule to 2358-2359 0 to force synch >> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >> cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >> >> tree,cn=config >> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica acquired >> successfully: Incremental update succeeded: start: 0: end: 0 >> Background task created to clean replication data. This may take a while. >> This may be safely interrupted with Ctrl+C >> >> ^CWait for task interrupted. It will continue to run in the background >> >> Deleted replication agreement from 'vm-055.idm.lab.bos.redhat.com' to >> 'vm-072.idm.lab.bos.redhat.com' >> ipa: INFO: Setting agreement >> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >> >> tree,cn=config schedule to 2358-2359 0 to force synch >> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >> >> tree,cn=config >> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica acquired >> successfully: Incremental update succeeded: start: 0: end: 0 >> Background task created to clean replication data. This may take a while. >> This may be safely interrupted with Ctrl+C >> >> ^CWait for task interrupted. It will continue to run in the background >> >> Deleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to >> 'vm-072.idm.lab.bos.redhat.com' >> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record does not >> contain 'vm-072.idm.lab.bos.redhat.com.' >> You may need to manually remove them from the tree >> >> Martin >> > > All issues addressed and I pulled in abort-clean-ruv from 1050. I added a > list-clean-ruv command as well. > > rob 1) Patch 1031-9 needs to get squashed with 1031-8 2) Patch needs a rebase (conflict in freeipa.spec.in) 3) New list-clean-ruv man entry is not right: list-clean-ruv [REPLICATION_ID] - List all running CLEANALLRUV and abort CLEANALLRUV tasks. REPLICATION_ID is not its argument. Btw. new list-clean-ruv command proved very useful for me. 4) I just found out we need to do a better job with make_readonly() command. I get into trouble when disconnecting one link to a remote replica as it was marked readonly and then I was then unable to manage the disconnected replica properly (vm-072 is the replica made readonly): [root at vm-055 ~]# ipa-replica-manage disconnect vm-072.idm.lab.bos.redhat.com [root at vm-072 ~]# ipa-replica-manage del vm-055.idm.lab.bos.redhat.com Deleting a master is irreversible. To reconnect to the remote master you will need to prepare a new replica file and re-install. Continue to delete? [no]: yes Deleting replication agreements between vm-055.idm.lab.bos.redhat.com and vm-072.idm.lab.bos.redhat.com ipa: INFO: Setting agreement cn=meTovm-072.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping tree,cn=config schedule to 2358-2359 0 to force synch ipa: INFO: Deleting schedule 2358-2359 0 from agreement cn=meTovm-072.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping tree,cn=config ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica acquired successfully: Incremental update succeeded: start: 0: end: 0 Deleted replication agreement from 'vm-072.idm.lab.bos.redhat.com' to 'vm-055.idm.lab.bos.redhat.com' Unable to remove replication agreement for vm-055.idm.lab.bos.redhat.com from vm-072.idm.lab.bos.redhat.com. Background task created to clean replication data. This may take a while. This may be safely interrupted with Ctrl+C ^CWait for task interrupted. It will continue to run in the background Failed to cleanup vm-055.idm.lab.bos.redhat.com entries: Server is unwilling to perform: database is read-only arguments: dn=krbprincipalname=ldap/vm-055.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM,cn=services,cn=accounts,dc=idm,dc=lab,dc=bos,dc=redhat,dc=com You may need to manually remove them from the tree ipa: INFO: Unhandled LDAPError: {'info': 'database is read-only', 'desc': 'Server is unwilling to perform'} Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: Server is unwilling to perform: database is read-only You may need to manually remove them from the tree --cleanup did not work for me as well: [root at vm-072 ~]# ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force --cleanup Cleaning a master is irreversible. This should not normally be require, so use cautiously. Continue to clean master? [no]: yes unexpected error: Server is unwilling to perform: database is read-only arguments: dn=krbprincipalname=ldap/vm-055.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM,cn=services,cn=accounts,dc=idm,dc=lab,dc=bos,dc=redhat,dc=com Martin From pspacek at redhat.com Mon Sep 17 09:18:53 2012 From: pspacek at redhat.com (Petr Spacek) Date: Mon, 17 Sep 2012 11:18:53 +0200 Subject: [Freeipa-devel] IPA server resolv.conf In-Reply-To: <5056CE2B.4060407@redhat.com> References: <1347747296.19861.78.camel@willson.li.ssimo.org> <5056CBE1.3000100@redhat.com> <5056CE2B.4060407@redhat.com> Message-ID: <5056EAFD.7040709@redhat.com> On 09/17/2012 09:15 AM, Martin Kosek wrote: > On 09/17/2012 09:06 AM, Petr Spacek wrote: >> Discussion about patch "Set master_kdc and dns_lookup_kdc to true)" reminds one >> related problem: >> >> Our server installer puts line "nameserver 127.0.0.1" to /etc/resolv.conf, but >> this file should contain all (or three nearest) DNS servers in IPA domain. >> >> As a result, IPA server will work even after local named crash (which is not so >> rare as I want :-(). >> >> New ticket: >> https://fedorahosted.org/freeipa/ticket/3085 >> >> Martin, what do you think? >> >> How we can update resolv.conf to reflect replica addition/deletion? >> >> Should it be done manually? E.g. ipa-replica-install script can print "don't >> forget to add this server to /etc/resolv.conf on other servers"? >> >> Petr^2 Spacek >> > > It would not be difficult to pull a list of IPA masters with DNS support during > ipa-{server,replica}-install and write more IPs to the resolv.conf. But I think > there may be an issue when somebody willingly stop a remote replica or > uninstall it. He would also need to remove it's IP from all resolv.confs in all > replicas... > > Btw. why would IPA server fail when a local named crashes? A record in > /etc/hosts we always add should still enable local IPA services to work or do I > miss something? Well... try it :-D "service named stop" I didn't examine details of this problem, but my guess is Kerberos and reverse DNS lookups. Also, you need to resolve neighbouring replica IP and so on. Name servers listed in resolv.conf are tried in order, so 127.0.0.1 should be on first place. man resolv.conf: nameserver Name server IP address ... Up to MAXNS (currently 3, see ) name servers may be listed, one per keyword. If there are multiple servers, the resolver library queries them in the order listed. ... (The algorithm used is to try a name server, and if the query times out, try the next, until out of name servers, then repeat trying all the name servers until a maximum number of retries are made.) Also, some update mechanism for resolv.conf would be nice. We should provide "gen-recolv-conf.py script" at least, so admin can call it from cron or someting like that. Petr^2 Spacek From sbose at redhat.com Mon Sep 17 10:23:03 2012 From: sbose at redhat.com (Sumit Bose) Date: Mon, 17 Sep 2012 12:23:03 +0200 Subject: [Freeipa-devel] [PATCH] Set master_kdc and dns_lookup_kdc to true In-Reply-To: <1347747296.19861.78.camel@willson.li.ssimo.org> References: <20120907120201.GC30746@localhost.localdomain> <5053A843.7070005@redhat.com> <20120915200210.GE28400@localhost.localdomain> <1347747296.19861.78.camel@willson.li.ssimo.org> Message-ID: <20120917102303.GF28400@localhost.localdomain> On Sat, Sep 15, 2012 at 06:14:56PM -0400, Simo Sorce wrote: > On Sat, 2012-09-15 at 22:02 +0200, Sumit Bose wrote: > > On Fri, Sep 14, 2012 at 05:57:23PM -0400, Rob Crittenden wrote: > > > Sumit Bose wrote: > > > >Hi, > > > > > > > >those two patches should fix > > > >https://fedorahosted.org/freeipa/ticket/2515 . The first makes the > > > >needed change for fresh installations. The second adds the changes > > > >during ipa-adtrust-install if needed. I prefer to do the changes here > > > >instead of during updates, because during updates it is not easy to see > > > >that the Kerberos configuration was changes. > > > > > > > > > > I guess it is good form to update the RHEL 4 client installer but > > > will anyone test it? > > > > I think it would be confusion if the RHEL4 client installer has > > different information than the default one. > > > > > > > > Is master_kdc supported in the MIT kfw version (krb5.ini)? > > > > For me it looks that the parse is build from the same sources. > > > > > > > > This suffers from the problem Simo envisioned with ticket 931. If > > > the /etc/hosts entry is removed then DNS will not start. We add an > > > entry during installation, so this may be less of an issue. > > > > If the /etc/hosts entry is removed DNS will not start in either case. > > > > I think the solution to #931 is setting the master_kdc option. You can > > easily reproduce startup problems if you set 'dns_lookup_kdc = true', > > stop sssd and try to restart named. This will run into a timeout and > > bind will not start. The reason is that besides a KDC the Kerberos > > client libraries also try to look up the master KDC (but it seems to be > > ok if the lookup finally fails). If sssd is running the locator plugin > > will return the current KDC as master. If it is not running, as in the > > test described above, /etc/krb5.conf is used next. If it does not have a > > master_kdc entry and 'dns_lookup_kdc = false' there is no other source > > for the master KDC and the client libraries continue with normal > > processing. If master_kdc is not set but 'dns_lookup_kdc = true' then a > > DNS lookup is tried, which will run into a timeout since the DNS server > > is not started yet. But if master_kdc is set in krb5.conf the client > > libraries will just use this value and will not try any DNS lookup, > > independently of the setting of dns_lookup_kdc. > > > > As a side note. Since we run named as user named I wonder if it would be > > possible to use SASL EXTERNAL auth instead of GSSAPI to bind to the LDAP > > server. If this would work safe and secure there would be no > > dependencies to the KDC during the startup of bind? > > The reason why we use gssapi is so that all operations performed by bind > happen as the DNS/fqdn user, and we can use ACIs targeted at the bind > process. In order to use SASL EXTERNAL we would need the bind process to > change euid to an unprivileged user that we then need to map to some > specific user. As said above named is already run as the unprivileged user named. > > In general krb5kdc should always start before named, and should not > depend on DNS resolution. If krb5kdc is started first bind should have > no issues. The only proble is if gssapi libraries try to use DNS > resolution, but we should have that solved by using the krb locator > plugin. yes, and even if the locator plugin isn't available setting master_kdc will make sure we never fall back to DNS for the local realm. Just to make sure, I do not want to say that the authentication type used by named must be changes to solve potential issues. Setting master_kdc will solve them. bye, Sumit > > Simo. > > -- > Simo Sorce * Red Hat, Inc * New York > From sbose at redhat.com Mon Sep 17 10:34:35 2012 From: sbose at redhat.com (Sumit Bose) Date: Mon, 17 Sep 2012 12:34:35 +0200 Subject: [Freeipa-devel] IPA server resolv.conf In-Reply-To: <5056EAFD.7040709@redhat.com> References: <1347747296.19861.78.camel@willson.li.ssimo.org> <5056CBE1.3000100@redhat.com> <5056CE2B.4060407@redhat.com> <5056EAFD.7040709@redhat.com> Message-ID: <20120917103435.GG28400@localhost.localdomain> On Mon, Sep 17, 2012 at 11:18:53AM +0200, Petr Spacek wrote: > On 09/17/2012 09:15 AM, Martin Kosek wrote: > >On 09/17/2012 09:06 AM, Petr Spacek wrote: > >>Discussion about patch "Set master_kdc and dns_lookup_kdc to true)" reminds one > >>related problem: > >> > >>Our server installer puts line "nameserver 127.0.0.1" to /etc/resolv.conf, but > >>this file should contain all (or three nearest) DNS servers in IPA domain. > >> > >>As a result, IPA server will work even after local named crash (which is not so > >>rare as I want :-(). > >> > >>New ticket: > >>https://fedorahosted.org/freeipa/ticket/3085 > >> > >>Martin, what do you think? > >> > >>How we can update resolv.conf to reflect replica addition/deletion? > >> > >>Should it be done manually? E.g. ipa-replica-install script can print "don't > >>forget to add this server to /etc/resolv.conf on other servers"? > >> > >>Petr^2 Spacek > >> > > > >It would not be difficult to pull a list of IPA masters with DNS support during > >ipa-{server,replica}-install and write more IPs to the resolv.conf. But I think > >there may be an issue when somebody willingly stop a remote replica or > >uninstall it. He would also need to remove it's IP from all resolv.confs in all > >replicas... > > > >Btw. why would IPA server fail when a local named crashes? A record in > >/etc/hosts we always add should still enable local IPA services to work or do I > >miss something? > > Well... try it :-D "service named stop" > > I didn't examine details of this problem, but my guess is Kerberos > and reverse DNS lookups. Also, you need to resolve neighbouring at least reverse DNS lookups shouldn't be the case since 'rdns = false' in krb5.conf. bye, Sumit > replica IP and so on. > > > Name servers listed in resolv.conf are tried in order, so 127.0.0.1 > should be on first place. > > man resolv.conf: > nameserver Name server IP address > ... Up to MAXNS (currently 3, see ) name servers > may be listed, one per keyword. If there are multiple servers, > the resolver library queries them in the order listed. > ... > (The algorithm used is to try a name server, and if the query times > out, try the next, until out of name servers, then repeat trying all > the name servers until a maximum number of retries are made.) > > > Also, some update mechanism for resolv.conf would be nice. We should > provide "gen-recolv-conf.py script" at least, so admin can call it > from cron or someting like that. > > Petr^2 Spacek > > _______________________________________________ > Freeipa-devel mailing list > Freeipa-devel at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-devel From pviktori at redhat.com Mon Sep 17 11:45:44 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Mon, 17 Sep 2012 13:45:44 +0200 Subject: [Freeipa-devel] [PATCH] 0077 Check direct/reverse hostname/address resolution in ipa-replica-install In-Reply-To: <5052D2CC.2000405@redhat.com> References: <5046009E.2020306@redhat.com> <50463DE6.9070300@redhat.com> <5046FF0E.1000504@redhat.com> <504FA7A3.5080201@redhat.com> <5050B1ED.70307@redhat.com> <50524387.4020800@redhat.com> <5052D2CC.2000405@redhat.com> Message-ID: <50570D68.9040501@redhat.com> On 09/14/2012 08:46 AM, Martin Kosek wrote: > On 09/13/2012 10:35 PM, Rob Crittenden wrote: >> Petr Viktorin wrote: >>> On 09/11/2012 11:05 PM, Rob Crittenden wrote: >>>> Petr Viktorin wrote: >>>>> On 09/04/2012 07:44 PM, Rob Crittenden wrote: >>>>>> Petr Viktorin wrote: >>>>>>> >>>>>>> https://fedorahosted.org/freeipa/ticket/2845 >>>>>> >>>>>> Shouldn't this also call verify_fqdn() on the local hostname and not >>>>>> just the master? I think this would eventually fail in the conncheck >>>>>> but >>>>>> what if that was skipped? >>>>>> >>>>>> rob >>>>> >>>>> A few lines above there is a call to get_host_name, which will call >>>>> verify_fqdn. >>>>> >>>> >>>> I double-checked this, it fails in conncheck. Here are my steps: >>>> >>>> # ipa-server-install --setup-dns >>>> # ipa-replica-prepare replica.example.com --ip-address=192.168.100.2 >>>> # ipa host-del replica.example.com >>>> >>>> On replica, set DNS to IPA master, with hostname in /etc/hosts. >>>> >>>> # ipa-replica-install ... >>>> >>>> The verify_fqdn() passes because the resolver uses /etc/hosts. >>>> >>>> The conncheck fails: >>>> >>>> Execute check on remote master >>>> Check connection from master to remote replica 'replica.example.com': >>>> >>>> Remote master check failed with following error message(s): >>>> Could not chdir to home directory /home/admin: No such file or directory >>>> Port check failed! Unable to resolve host name 'replica.example.com' >>>> >>>> Connection check failed! >>>> Please fix your network settings according to error messages above. >>>> If the check results are not valid it can be skipped with >>>> --skip-conncheck parameter. >>>> >>>> The DNS test happens much further after this, and I get why, I just >>>> don't see how useful it is unless the --skip-conncheck is used. >>> >>> For the record, it's because we need to check if the host has DNS >>> installed. We need a LDAP connection to check this. >>> >>>> ipa-replica-install ~rcrit/replica-info-replica.example.com.gpg >>>> --skip-conncheck >>>> Directory Manager (existing master) password: >>>> >>>> ipa : ERROR Could not resolve hostname replica.example.com >>>> using DNS. Clients may not function properly. Please check your DNS >>>> setup. (Note that this check queries IPA DNS directly and ignores >>>> /etc/hosts.) >>>> Continue? [no]: >>>> >>>> So I guess, what are the intentions here? It is certainly better than >>>> before. >>>> >>>> rob >>> >>> If the replica is in the master's /etc/hosts, but not in DNS, the >>> conncheck will succeed. This check explicitly queries IPA records only >>> and ignores /etc/hosts so it'll notice this case and warn. >>> >> >> Ok, like I said, this is better than we have. Just one nit then you get an ack: >> >> + # If remote host has DNS, check forward/reverse resolution >> + try: >> + entry = conn.find_entries(u'cn=dns', base_dn=DN(api.env.basedn)) >> + except errors.NotFound: >> >> u'cn=dns' should be str(constants.container_dns). >> >> rob > > This is a search filter, Petr could use the one I already have in > "dns.py::get_dns_masters()" function: > '(&(objectClass=ipaConfigObject)(cn=DNS))' > > For performance sake, I would also not search in the entire tree, but limit the > search only to: > > DN(('cn', 'masters'), ('cn', 'ipa'), ('cn', 'etc'), api.env.basedn) > > Martin > Attaching updated patch with Martin's suggestions. -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0077-02-Check-direct-reverse-hostname-address-resolution-in-.patch Type: text/x-patch Size: 5926 bytes Desc: not available URL: From pviktori at redhat.com Mon Sep 17 12:32:57 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Mon, 17 Sep 2012 14:32:57 +0200 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <50539F44.5040308@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> <1347329065.24731.101.camel@aleeredhat.laptop> <504F318F.3030603@redhat.com> <504F35AD.3030905@redhat.com> <1347373691.24731.125.camel@aleeredhat.laptop> <504F4CE0.9080706@redhat.com> <504F67C5.4000600@redhat.com> <504F86C0.6060907@redhat.com> <5050BB14.3050502@redhat.com> <5053350D.5010805@redhat.com> <50539F44.5040308@redhat.com> Message-ID: <50571879.3080401@redhat.com> On 09/14/2012 11:19 PM, Rob Crittenden wrote: > Petr Viktorin wrote: >> On 09/12/2012 06:40 PM, Petr Viktorin wrote: >>> A new Dogtag build with changed pkispawn/pkidestroy locations should be >>> out later today. The attached patch should work with that build. > > Fresh install is failing in F-18. > > ki-tools-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.i686 > pki-base-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.noarch > pki-server-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.noarch > pki-silent-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.noarch > pki-symkey-9.0.21-1.fc18.x86_64 > dogtag-pki-ca-theme-10.0.0-0.1.a1.20120914T0604zgit69c0684.fc18.noarch > pki-selinux-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.noarch > pki-ca-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.noarch > pki-setup-9.0.21-1.fc18.noarch > > > rob > > Ade, your patch adds a step of moving /var/lib/pki/pki-tomcat/alias/ca_admin_cert.p12 to /root/ca-agent.p12 right after calling pkispawn. It seems the file is not created on f18. Did something change in Dogtag or are we calling it incorrectly? -- Petr? From pspacek at redhat.com Mon Sep 17 12:55:06 2012 From: pspacek at redhat.com (Petr Spacek) Date: Mon, 17 Sep 2012 14:55:06 +0200 Subject: [Freeipa-devel] [PATCH 0063] Notify DNS slaves if zone serial number modification was detected. Message-ID: <50571DAA.5030506@redhat.com> Hello, this patch adds missing notification to DNS slaves if zone serial number modification was detected. Petr^2 Spacek -------------- next part -------------- A non-text attachment was scrubbed... Name: bind-dyndb-ldap-pspacek-0063-Notify-DNS-slaves-if-zone-serial-number-modification.patch Type: text/x-patch Size: 3074 bytes Desc: not available URL: From rcritten at redhat.com Mon Sep 17 14:04:31 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 17 Sep 2012 10:04:31 -0400 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <5056E754.1080201@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> <50479491.1060200@redhat.com> <50488DD0.1080507@redhat.com> <5048BEF8.5000401@redhat.com> <5048C46A.8020207@redhat.com> <5048C754.2030708@redhat.com> <5048C9BE.1050005@redhat.com> <504912F9.1020907@redhat.com> <50531DDC.5050007@redhat.com> <505382CF.1030901@redhat.com> <5056E754.1080201@redhat.com> Message-ID: <50572DEF.7020104@redhat.com> Martin Kosek wrote: > On 09/14/2012 09:17 PM, Rob Crittenden wrote: >> Martin Kosek wrote: >>> On 09/06/2012 11:17 PM, Rob Crittenden wrote: >>>> Martin Kosek wrote: >>>>> On 09/06/2012 05:55 PM, Rob Crittenden wrote: >>>>>> Rob Crittenden wrote: >>>>>>> Rob Crittenden wrote: >>>>>>>> Martin Kosek wrote: >>>>>>>>> On 09/05/2012 08:06 PM, Rob Crittenden wrote: >>>>>>>>>> Rob Crittenden wrote: >>>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>>> On 07/05/2012 08:39 PM, Rob Crittenden wrote: >>>>>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>>>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: >>>>>>>>>>>>>>> Deleting a replica can leave a replication vector (RUV) on the >>>>>>>>>>>>>>> other servers. >>>>>>>>>>>>>>> This can confuse things if the replica is re-added, and it also >>>>>>>>>>>>>>> causes the >>>>>>>>>>>>>>> server to calculate changes against a server that may no longer >>>>>>>>>>>>>>> exist. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 389-ds-base provides a new task that self-propogates itself to all >>>>>>>>>>>>>>> available >>>>>>>>>>>>>>> replicas to clean this RUV data. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This patch will create this task at deletion time to hopefully >>>>>>>>>>>>>>> clean things up. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It isn't perfect. If any replica is down or unavailable at the >>>>>>>>>>>>>>> time >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> cleanruv task fires, and then comes back up, the old RUV data >>>>>>>>>>>>>>> may be >>>>>>>>>>>>>>> re-propogated around. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> To make things easier in this case I've added two new commands to >>>>>>>>>>>>>>> ipa-replica-manage. The first lists the replication ids of all the >>>>>>>>>>>>>>> servers we >>>>>>>>>>>>>>> have a RUV for. Using this you can call clean_ruv with the >>>>>>>>>>>>>>> replication id of a >>>>>>>>>>>>>>> server that no longer exists to try the cleanallruv step again. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is quite dangerous though. If you run cleanruv against a >>>>>>>>>>>>>>> replica id that >>>>>>>>>>>>>>> does exist it can cause a loss of data. I believe I've put in >>>>>>>>>>>>>>> enough scary >>>>>>>>>>>>>>> warnings about this. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> rob >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Good work there, this should make cleaning RUVs much easier than >>>>>>>>>>>>>> with the >>>>>>>>>>>>>> previous version. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is what I found during review: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1) list_ruv and clean_ruv command help in man is quite lost. I >>>>>>>>>>>>>> think >>>>>>>>>>>>>> it would >>>>>>>>>>>>>> help if we for example have all info for commands indented. This >>>>>>>>>>>>>> way >>>>>>>>>>>>>> user could >>>>>>>>>>>>>> simply over-look the new commands in the man page. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2) I would rename new commands to clean-ruv and list-ruv to make >>>>>>>>>>>>>> them >>>>>>>>>>>>>> consistent with the rest of the commands (re-initialize, >>>>>>>>>>>>>> force-sync). >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 3) It would be nice to be able to run clean_ruv command in an >>>>>>>>>>>>>> unattended way >>>>>>>>>>>>>> (for better testing), i.e. respect --force option as we already >>>>>>>>>>>>>> do for >>>>>>>>>>>>>> ipa-replica-manage del. This fix would aid test automation in the >>>>>>>>>>>>>> future. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 4) (minor) The new question (and the del too) does not react too >>>>>>>>>>>>>> well for >>>>>>>>>>>>>> CTRL+D: >>>>>>>>>>>>>> >>>>>>>>>>>>>> # ipa-replica-manage clean_ruv 3 --force >>>>>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>>>>> Continue to clean? [no]: unexpected error: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 5) Help for clean_ruv command without a required parameter is quite >>>>>>>>>>>>>> confusing >>>>>>>>>>>>>> as it reports that command is wrong and not the parameter: >>>>>>>>>>>>>> >>>>>>>>>>>>>> # ipa-replica-manage clean_ruv >>>>>>>>>>>>>> Usage: ipa-replica-manage [options] >>>>>>>>>>>>>> >>>>>>>>>>>>>> ipa-replica-manage: error: must provide a command [clean_ruv | >>>>>>>>>>>>>> force-sync | >>>>>>>>>>>>>> disconnect | connect | del | re-initialize | list | list_ruv] >>>>>>>>>>>>>> >>>>>>>>>>>>>> It seems you just forgot to specify the error message in the >>>>>>>>>>>>>> command >>>>>>>>>>>>>> definition >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 6) When the remote replica is down, the clean_ruv command fails >>>>>>>>>>>>>> with an >>>>>>>>>>>>>> unexpected error: >>>>>>>>>>>>>> >>>>>>>>>>>>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 >>>>>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>>>>> Continue to clean? [no]: y >>>>>>>>>>>>>> unexpected error: {'desc': 'Operations error'} >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: >>>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>>> cleanAllRUV_task: failed >>>>>>>>>>>>>> to connect to repl agreement connection >>>>>>>>>>>>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, >>>>>>>>>>>>>> >>>>>>>>>>>>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> tree,cn=config), error 105 >>>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>>> cleanAllRUV_task: replica >>>>>>>>>>>>>> (cn=meTovm-055.idm.lab. >>>>>>>>>>>>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> tree, cn=config) has not been cleaned. You will need to rerun >>>>>>>>>>>>>> the >>>>>>>>>>>>>> CLEANALLRUV task on this replica. >>>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>>> cleanAllRUV_task: Task >>>>>>>>>>>>>> failed (1) >>>>>>>>>>>>>> >>>>>>>>>>>>>> In this case I think we should inform user that the command failed, >>>>>>>>>>>>>> possibly >>>>>>>>>>>>>> because of disconnected replicas and that they could enable the >>>>>>>>>>>>>> replicas and >>>>>>>>>>>>>> try again. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 7) (minor) "pass" is now redundant in replication.py: >>>>>>>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>>>>>>>> + # We can't make the server we're removing read-only >>>>>>>>>>>>>> but >>>>>>>>>>>>>> + # this isn't a show-stopper >>>>>>>>>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>>>>>>>>> read-only, >>>>>>>>>>>>>> continuing anyway") >>>>>>>>>>>>>> + pass >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I think this addresses everything. >>>>>>>>>>>>> >>>>>>>>>>>>> rob >>>>>>>>>>>> >>>>>>>>>>>> Thanks, almost there! I just found one more issue which needs to be >>>>>>>>>>>> fixed >>>>>>>>>>>> before we push: >>>>>>>>>>>> >>>>>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>>>>>>>>>> Directory Manager password: >>>>>>>>>>>> >>>>>>>>>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing >>>>>>>>>>>> removal >>>>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>>>>>> "Can't >>>>>>>>>>>> contact LDAP server"} >>>>>>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>>>>>> >>>>>>>>>>>> There were issues removing a connection: %d format: a number is >>>>>>>>>>>> required, not str >>>>>>>>>>>> >>>>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>>>>>> "Can't >>>>>>>>>>>> contact LDAP server"} >>>>>>>>>>>> >>>>>>>>>>>> This is a traceback I retrieved: >>>>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>>> File "/sbin/ipa-replica-manage", line 425, in del_master >>>>>>>>>>>> del_link(realm, r, hostname, options.dirman_passwd, force=True) >>>>>>>>>>>> File "/sbin/ipa-replica-manage", line 271, in del_link >>>>>>>>>>>> repl1.cleanallruv(replica_id) >>>>>>>>>>>> File >>>>>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>>>>>>>>> line 1094, in cleanallruv >>>>>>>>>>>> root_logger.debug("Creating CLEANALLRUV task for replica id >>>>>>>>>>>> %d" % >>>>>>>>>>>> replicaId) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> The problem here is that you don't convert replica_id to int in this >>>>>>>>>>>> part: >>>>>>>>>>>> + replica_id = None >>>>>>>>>>>> + if repl2: >>>>>>>>>>>> + replica_id = repl2._get_replica_id(repl2.conn, None) >>>>>>>>>>>> + else: >>>>>>>>>>>> + servers = get_ruv(realm, replica1, dirman_passwd) >>>>>>>>>>>> + for (netloc, rid) in servers: >>>>>>>>>>>> + if netloc.startswith(replica2): >>>>>>>>>>>> + replica_id = rid >>>>>>>>>>>> + break >>>>>>>>>>>> >>>>>>>>>>>> Martin >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Updated patch using new mechanism in 389-ds-base. This should more >>>>>>>>>>> thoroughly clean out RUV data when a replica is being deleted, and >>>>>>>>>>> provide for a way to delete RUV data afterwards too if necessary. >>>>>>>>>>> >>>>>>>>>>> rob >>>>>>>>>> >>>>>>>>>> Rebased patch >>>>>>>>>> >>>>>>>>>> rob >>>>>>>>>> >>>>>>>>> >>>>>>>>> 0) As I wrote in a review for your patch 1041, changelog entry slipped >>>>>>>>> elsewhere. >>>>>>>>> >>>>>>>>> 1) The following KeyboardInterrupt except class looks suspicious. I >>>>>>>>> know why >>>>>>>>> you have it there, but since it is generally a bad thing to do, some >>>>>>>>> comment >>>>>>>>> why it is needed would be useful. >>>>>>>>> >>>>>>>>> @@ -256,6 +263,17 @@ def del_link(realm, replica1, replica2, >>>>>>>>> dirman_passwd, >>>>>>>>> force=False): >>>>>>>>> repl1.delete_agreement(replica2) >>>>>>>>> repl1.delete_referral(replica2) >>>>>>>>> >>>>>>>>> + if type1 == replication.IPA_REPLICA: >>>>>>>>> + if repl2: >>>>>>>>> + ruv = repl2._get_replica_id(repl2.conn, None) >>>>>>>>> + else: >>>>>>>>> + ruv = get_ruv_by_host(realm, replica1, replica2, >>>>>>>>> dirman_passwd) >>>>>>>>> + >>>>>>>>> + try: >>>>>>>>> + repl1.cleanallruv(ruv) >>>>>>>>> + except KeyboardInterrupt: >>>>>>>>> + pass >>>>>>>>> + >>>>>>>>> >>>>>>>>> Maybe you just wanted to do some cleanup and then "raise" again? >>>>>>>> >>>>>>>> No, it is there because it is safe to break out of it. The task will >>>>>>>> continue to run. I added some verbiage. >>>>>>>> >>>>>>>>> >>>>>>>>> 2) This is related to 1), but when some replica is down, >>>>>>>>> "ipa-replica-manage >>>>>>>>> del" may wait indefinitely when some remote replica is down, right? >>>>>>>>> >>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>>>>> Deleting a master is irreversible. >>>>>>>>> To reconnect to the remote master you will need to prepare a new >>>>>>>>> replica file >>>>>>>>> and re-install. >>>>>>>>> Continue to delete? [no]: y >>>>>>>>> ipa: INFO: Setting agreement >>>>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>>>>>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> tree,cn=config >>>>>>>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica >>>>>>>>> acquired >>>>>>>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>>>>>>> Background task created to clean replication data >>>>>>>>> >>>>>>>>> ... after about a minute I hit CTRL+C >>>>>>>>> >>>>>>>>> ^CDeleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to >>>>>>>>> 'vm-055.idm.lab.bos.redhat.com' >>>>>>>>> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: NS record >>>>>>>>> does not >>>>>>>>> contain 'vm-055.idm.lab.bos.redhat.com.' >>>>>>>>> You may need to manually remove them from the tree >>>>>>>>> >>>>>>>>> I think it would be better to inform user that some remote replica is >>>>>>>>> down or >>>>>>>>> at least that we are waiting for the task to complete. Something like >>>>>>>>> that: >>>>>>>>> >>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>>>>> ... >>>>>>>>> Background task created to clean replication data >>>>>>>>> Replication data clean up may take very long time if some replica is >>>>>>>>> unreachable >>>>>>>>> Hit CTRL+C to interrupt the wait >>>>>>>>> ^C Clean up wait interrupted >>>>>>>>> .... >>>>>>>>> [continue with del] >>>>>>>> >>>>>>>> Yup, did this in #1. >>>>>>>> >>>>>>>>> >>>>>>>>> 3) (minor) When there is a cleanruv task running and you run >>>>>>>>> "ipa-replica-manage del", there is a unexpected error message with >>>>>>>>> duplicate >>>>>>>>> task object in LDAP: >>>>>>>>> >>>>>>>>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com --force >>>>>>>>> Unable to connect to replica vm-072.idm.lab.bos.redhat.com, forcing >>>>>>>>> removal >>>>>>>>> FAIL >>>>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>>>>>> contact LDAP server"} >>>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>>> >>>>>>>>> There were issues removing a connection: This entry already exists >>>>>>>>> <<<<<<<<< >>>>>>>>> >>>>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>>>>>> contact LDAP server"} >>>>>>>>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record >>>>>>>>> does not >>>>>>>>> contain 'vm-072.idm.lab.bos.redhat.com.' >>>>>>>>> You may need to manually remove them from the tree >>>>>>>>> >>>>>>>>> >>>>>>>>> I think it should be enough to just catch for "entry already exists" in >>>>>>>>> cleanallruv function, and in such case print a relevant error message >>>>>>>>> bail out. >>>>>>>>> Thus, self.conn.checkTask(dn, dowait=True) would not be called too. >>>>>>>> >>>>>>>> Good catch, fixed. >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 4) (minor): In make_readonly function, there is a redundant "pass" >>>>>>>>> statement: >>>>>>>>> >>>>>>>>> + def make_readonly(self): >>>>>>>>> + """ >>>>>>>>> + Make the current replication agreement read-only. >>>>>>>>> + """ >>>>>>>>> + dn = DN(('cn', 'userRoot'), ('cn', 'ldbm database'), >>>>>>>>> + ('cn', 'plugins'), ('cn', 'config')) >>>>>>>>> + >>>>>>>>> + mod = [(ldap.MOD_REPLACE, 'nsslapd-readonly', 'on')] >>>>>>>>> + try: >>>>>>>>> + self.conn.modify_s(dn, mod) >>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>>> + # We can't make the server we're removing read-only but >>>>>>>>> + # this isn't a show-stopper >>>>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>>>> read-only, >>>>>>>>> continuing anyway") >>>>>>>>> + pass <<<<<<<<<<<<<<< >>>>>>>> >>>>>>>> Yeah, this is one of my common mistakes. I put in a pass initially, then >>>>>>>> add logging in front of it and forget to delete the pass. Its gone now. >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 5) In clean_ruv, I think allowing a --force option to bypass the >>>>>>>>> user_input >>>>>>>>> would be helpful (at least for test automation): >>>>>>>>> >>>>>>>>> + if not ipautil.user_input("Continue to clean?", False): >>>>>>>>> + sys.exit("Aborted") >>>>>>>> >>>>>>>> Yup, added. >>>>>>>> >>>>>>>> rob >>>>>>> >>>>>>> Slightly revised patch. I still had a window open with one unsaved change. >>>>>>> >>>>>>> rob >>>>>>> >>>>>> >>>>>> Apparently there were two unsaved changes, one of which was lost. This >>>>>> adds in >>>>>> the 'entry already exists' fix. >>>>>> >>>>>> rob >>>>>> >>>>> >>>>> Just one last thing (otherwise the patch is OK) - I don't think this is >>>>> what we >>>>> want :-) >>>>> >>>>> # ipa-replica-manage clean-ruv 8 >>>>> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 >>>>> >>>>> Cleaning the wrong replica ID will cause that server to no >>>>> longer replicate so it may miss updates while the process >>>>> is running. It would need to be re-initialized to maintain >>>>> consistency. Be very careful. >>>>> Continue to clean? [no]: y <<<<<< >>>>> Aborted >>>>> >>>>> >>>>> Nor this exception, (your are checking for wrong exception): >>>>> >>>>> # ipa-replica-manage clean-ruv 8 >>>>> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 >>>>> >>>>> Cleaning the wrong replica ID will cause that server to no >>>>> longer replicate so it may miss updates while the process >>>>> is running. It would need to be re-initialized to maintain >>>>> consistency. Be very careful. >>>>> Continue to clean? [no]: >>>>> unexpected error: This entry already exists >>>>> >>>>> This is the exception: >>>>> >>>>> Traceback (most recent call last): >>>>> File "/sbin/ipa-replica-manage", line 651, in >>>>> main() >>>>> File "/sbin/ipa-replica-manage", line 648, in main >>>>> clean_ruv(realm, args[1], options) >>>>> File "/sbin/ipa-replica-manage", line 373, in clean_ruv >>>>> thisrepl.cleanallruv(ruv) >>>>> File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>> line 1136, in cleanallruv >>>>> self.conn.addEntry(e) >>>>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 503, in >>>>> addEntry >>>>> self.__handle_errors(e, arg_desc=arg_desc) >>>>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line 321, in >>>>> __handle_errors >>>>> raise errors.DuplicateEntry() >>>>> ipalib.errors.DuplicateEntry: This entry already exists >>>>> >>>>> Martin >>>>> >>>> >>>> Fixed that and a couple of other problems. When doing a disconnect we should >>>> not also call clean-ruv. >>> >>> Ah, good self-catch. >>> >>>> >>>> I also got tired of seeing crappy error messages so I added a little convert >>>> utility. >>>> >>>> rob >>> >>> 1) There is CLEANALLRUV stuff included in 1050-3 and not here. There are also >>> some finding for this new code. >>> >>> >>> 2) We may want to bump Requires to higher version of 389-ds-base >>> (389-ds-base-1.2.11.14-1) - it contains a fix for CLEANALLRUV+winsync bug I >>> found earlier. >>> >>> >>> 3) I just discovered another suspicious behavior. When we are deleting a master >>> that has links also to other master(s) we delete those too. But we also >>> automatically run CLEANALLRUV in these cases, so we may end up in multiple >>> tasks being started on different masters - this does not look right. >>> >>> I think we may rather want to at first delete all links and then run >>> CLEANALLRUV task, just for one time. This is what I get with current code: >>> >>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com >>> Directory Manager password: >>> >>> Deleting a master is irreversible. >>> To reconnect to the remote master you will need to prepare a new replica file >>> and re-install. >>> Continue to delete? [no]: yes >>> ipa: INFO: Setting agreement >>> cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>> >>> tree,cn=config schedule to 2358-2359 0 to force synch >>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>> cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>> >>> tree,cn=config >>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica acquired >>> successfully: Incremental update succeeded: start: 0: end: 0 >>> Background task created to clean replication data. This may take a while. >>> This may be safely interrupted with Ctrl+C >>> >>> ^CWait for task interrupted. It will continue to run in the background >>> >>> Deleted replication agreement from 'vm-055.idm.lab.bos.redhat.com' to >>> 'vm-072.idm.lab.bos.redhat.com' >>> ipa: INFO: Setting agreement >>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>> >>> tree,cn=config schedule to 2358-2359 0 to force synch >>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>> >>> tree,cn=config >>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica acquired >>> successfully: Incremental update succeeded: start: 0: end: 0 >>> Background task created to clean replication data. This may take a while. >>> This may be safely interrupted with Ctrl+C >>> >>> ^CWait for task interrupted. It will continue to run in the background >>> >>> Deleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to >>> 'vm-072.idm.lab.bos.redhat.com' >>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record does not >>> contain 'vm-072.idm.lab.bos.redhat.com.' >>> You may need to manually remove them from the tree >>> >>> Martin >>> >> >> All issues addressed and I pulled in abort-clean-ruv from 1050. I added a >> list-clean-ruv command as well. >> >> rob > > 1) Patch 1031-9 needs to get squashed with 1031-8 > > > 2) Patch needs a rebase (conflict in freeipa.spec.in) > > > 3) New list-clean-ruv man entry is not right: > > list-clean-ruv [REPLICATION_ID] > - List all running CLEANALLRUV and abort CLEANALLRUV tasks. > > REPLICATION_ID is not its argument. Fixed 1-3. > Btw. new list-clean-ruv command proved very useful for me. > > 4) I just found out we need to do a better job with make_readonly() command. I > get into trouble when disconnecting one link to a remote replica as it was > marked readonly and then I was then unable to manage the disconnected replica > properly (vm-072 is the replica made readonly): Ok, I reset read-only after we delete the agreements. That fixed things up for me. I disconnected a replica and was able to modify entries on that replica afterwards. This affected the --cleanup command too, it would otherwise have succeeded I think. I tested with an A - B - C - A agreement loop. I disconnected A and C and confirmed I could still update entries on C. Then I deleted C, then B, and made sure output looked right, I could still manage entries, etc. rob > > [root at vm-055 ~]# ipa-replica-manage disconnect vm-072.idm.lab.bos.redhat.com > > [root at vm-072 ~]# ipa-replica-manage del vm-055.idm.lab.bos.redhat.com > Deleting a master is irreversible. > To reconnect to the remote master you will need to prepare a new replica file > and re-install. > Continue to delete? [no]: yes > Deleting replication agreements between vm-055.idm.lab.bos.redhat.com and > vm-072.idm.lab.bos.redhat.com > ipa: INFO: Setting agreement > cn=meTovm-072.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping > tree,cn=config schedule to 2358-2359 0 to force synch > ipa: INFO: Deleting schedule 2358-2359 0 from agreement > cn=meTovm-072.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping > tree,cn=config > ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica acquired > successfully: Incremental update succeeded: start: 0: end: 0 > Deleted replication agreement from 'vm-072.idm.lab.bos.redhat.com' to > 'vm-055.idm.lab.bos.redhat.com' > Unable to remove replication agreement for vm-055.idm.lab.bos.redhat.com from > vm-072.idm.lab.bos.redhat.com. > Background task created to clean replication data. This may take a while. > This may be safely interrupted with Ctrl+C > ^CWait for task interrupted. It will continue to run in the background > > Failed to cleanup vm-055.idm.lab.bos.redhat.com entries: Server is unwilling to > perform: database is read-only arguments: > dn=krbprincipalname=ldap/vm-055.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM,cn=services,cn=accounts,dc=idm,dc=lab,dc=bos,dc=redhat,dc=com > > You may need to manually remove them from the tree > ipa: INFO: Unhandled LDAPError: {'info': 'database is read-only', 'desc': > 'Server is unwilling to perform'} > > Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: Server is > unwilling to perform: database is read-only > > You may need to manually remove them from the tree > > > --cleanup did not work for me as well: > [root at vm-072 ~]# ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force > --cleanup > Cleaning a master is irreversible. > This should not normally be require, so use cautiously. > Continue to clean master? [no]: yes > unexpected error: Server is unwilling to perform: database is read-only > arguments: > dn=krbprincipalname=ldap/vm-055.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM,cn=services,cn=accounts,dc=idm,dc=lab,dc=bos,dc=redhat,dc=com > > Martin > -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-rcrit-1031-10-cleanruv.patch Type: text/x-diff Size: 9525 bytes Desc: not available URL: From mkosek at redhat.com Mon Sep 17 14:06:37 2012 From: mkosek at redhat.com (Martin Kosek) Date: Mon, 17 Sep 2012 16:06:37 +0200 Subject: [Freeipa-devel] [PATCH] 1050 prevent replica orphans In-Reply-To: <50538284.8090205@redhat.com> References: <5040F6E2.7060109@redhat.com> <5040F722.4060208@redhat.com> <5048BFAD.1070807@redhat.com> <50491414.2070504@redhat.com> <1347008704.2829.13.camel@priserak> <504E329D.6070905@redhat.com> <50531B7C.8070401@redhat.com> <50538284.8090205@redhat.com> Message-ID: <50572E6D.6070201@redhat.com> On 09/14/2012 09:16 PM, Rob Crittenden wrote: > Martin Kosek wrote: >> On 09/10/2012 08:34 PM, Rob Crittenden wrote: >>> Martin Kosek wrote: >>>> On Thu, 2012-09-06 at 17:22 -0400, Rob Crittenden wrote: >>>>> Martin Kosek wrote: >>>>>> On 08/31/2012 07:40 PM, Rob Crittenden wrote: >>>>>>> Rob Crittenden wrote: >>>>>>>> It was possible use ipa-replica-manage connect/disconnect/del to end up >>>>>>>> orphaning or or more IPA masters. This is an attempt to catch and >>>>>>>> prevent that case. >>>>>>>> >>>>>>>> I tested with this topology, trying to delete B. >>>>>>>> >>>>>>>> A <-> B <-> C >>>>>>>> >>>>>>>> I got here by creating B and C from A, connecting B to C then deleting >>>>>>>> the link from A to B, so it went from A -> B and A -> C to the above. >>>>>>>> >>>>>>>> What I do is look up the servers that the delete candidate host has >>>>>>>> connections to and see if we're the last link. >>>>>>>> >>>>>>>> I added an escape clause if there are only two masters. >>>>>>>> >>>>>>>> rob >>>>>>> >>>>>>> Oh, this relies on my cleanruv patch 1031. >>>>>>> >>>>>>> rob >>>>>>> >>>>>> >>>>>> 1) When I run ipa-replica-manage del --force to an already uninstalled host, >>>>>> the new code will prevent me the deletation because it cannot connect to >>>>>> it. It >>>>>> also crashes with UnboundLocalError: >>>>>> >>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>>>> >>>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing removal >>>>>> Traceback (most recent call last): >>>>>> File "/sbin/ipa-replica-manage", line 708, in >>>>>> main() >>>>>> File "/sbin/ipa-replica-manage", line 677, in main >>>>>> del_master(realm, args[1], options) >>>>>> File "/sbin/ipa-replica-manage", line 476, in del_master >>>>>> sys.exit("Failed read master data from '%s': %s" % (delrepl.hostname, >>>>>> str(e))) >>>>>> UnboundLocalError: local variable 'delrepl' referenced before assignment >>>>> >>>>> Fixed. >>>>> >>>>>> >>>>>> >>>>>> I also hit this error when removing a winsync replica. >>>>> >>>>> Fixed. >>>>> >>>>>> >>>>>> >>>>>> 2) As I wrote before, I think having --force option override the user >>>>>> inquiries >>>>>> would benefit test automation: >>>>>> >>>>>> + if not ipautil.user_input("Continue to delete?", False): >>>>>> + sys.exit("Aborted") >>>>> >>>>> Fixed. >>>>> >>>>>> >>>>>> >>>>>> 3) I don't think this code won't cover this topology: >>>>>> >>>>>> A - B - C - D - E >>>>>> >>>>>> It would allow you deleting a replica C even though it would separate A-B >>>>>> and >>>>>> D-E. Though we may not want to cover this situation now, what you got is >>>>>> definitely helping. >>>>> >>>>> I think you may be right. I only tested with 4 servers. With this B and >>>>> D would both still have 2 agreements so wouldn't be covered by the last >>>>> link test. >>>> >>>> Everything looks good now, so ACK. We just need to push it along with >>>> CLEANALLRUV patch. >>>> >>>> Martin >>>> >>> >>> Not to look a gift ACK In the mouth but here is a revised patch. I've added a >>> cleanup routine to remove an orphaned master properly. If you had tried the >>> mechanism I outlined in the man page it would have worked but was >>> less-than-complete. This way is better, just don't try it on a live master. >>> >>> I also added a cleanruv abort command, in case you want to kill an existing >>> task. >>> >>> rob >>> >> >> 1) CLEANRUV stuff should be in your patch 1031 and not here (but I will comment >> on the code in this mail since it is in this patch) >> >> >> 2) In new command defitinion: >> >> + "abort-clean-ruv":(1, 1, "Replica ID of to abort cleaning", ""), >> >> I miss error message in case REPLICA_ID is not passed, this way error message >> when I omit the parameter is confusing: >> >> # ipa-replica-manage abort-clean-ruv >> Usage: ipa-replica-manage [options] >> >> ipa-replica-manage: error: must provide a command [force-sync | clean-ruv | >> disconnect | connect | list-ruv | del | re-initialize | list | abort-clean-ruv] >> >> On another note, I am thinking about the new command(s). Since we now have >> abort-clean-ruv command, we may want to also implement list-clean-ruv commands >> in the future to see all CLEANALLRUV commands in process... Or we may enhance >> list-ruv to write a flag like [CLEANALLRUV in process] for RUV's for which >> CLEANALLRUV task is in process. >> >> >> 3) Will clean-ruv - abort-clean-ruv - clean-ruv sequence work? If the aborted >> CLEANALLRUV task stays in DIT, we may not be able to enter new CLEANALLRUV task >> because we always use the same DN... >> >> Btw. did abort CLEANALLRUV command worked for you? Mine seemed to be stuck on >> replicas that are down just like CLEANALLRUV command. I had then paralelly >> running CLEANALLRUV and ABORT CLEANALLRUV running for the same RUV ID. Then, it >> is unclear to me what this command is actually good for. >> >> >> 4) Man page now contains non-existent command: >> >> --- a/install/tools/man/ipa-replica-manage.1 >> +++ b/install/tools/man/ipa-replica-manage.1 >> @@ -42,12 +42,18 @@ Manages the replication agreements of an IPA server. >> \fBforce\-sync\fR >> \- Immediately flush any data to be replicated from a server specified with >> the \-\-from option >> .TP >> +\fBcleanup\fR >> +\- Remove leftover references to a deleted master. >> +.TP >> >> >> The cleanup procedure was implemented rather as an option to the del command >> than a separate command. >> >> >> 5) My favorite - new cleanup procedure user prompt miss the --force option >> useful for test automation >> >> + if not ipautil.user_input("Continue to clean master?", False): >> + sys.exit("Cleanup aborted") >> >> >> Otherwise the patch looks good. >> >> Martin >> > > I pulled the abort-ruv stuff out. It was just easier to stuff it in here than > rebasing back, but yeah, its cleaner this way. > > No need to check force for clean because it is already required (can't contact > the deleted master, it's gone). > > rob The patch works fine now. So ACK when the dependent patch 1031 is pushed. Martin From mkosek at redhat.com Mon Sep 17 14:11:38 2012 From: mkosek at redhat.com (Martin Kosek) Date: Mon, 17 Sep 2012 16:11:38 +0200 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <50572DEF.7020104@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> <50479491.1060200@redhat.com> <50488DD0.1080507@redhat.com> <5048BEF8.5000401@redhat.com> <5048C46A.8020207@redhat.com> <5048C754.2030708@redhat.com> <5048C9BE.1050005@redhat.com> <504912F9.1020907@redhat.com> <50531DDC.5050007@redhat.com> <505382CF.1030901@redhat.com> <5056E754.1080201@redhat.com> <50572DEF.7020104@redhat.com> Message-ID: <50572F9A.3040605@redhat.com> On 09/17/2012 04:04 PM, Rob Crittenden wrote: > Martin Kosek wrote: >> On 09/14/2012 09:17 PM, Rob Crittenden wrote: >>> Martin Kosek wrote: >>>> On 09/06/2012 11:17 PM, Rob Crittenden wrote: >>>>> Martin Kosek wrote: >>>>>> On 09/06/2012 05:55 PM, Rob Crittenden wrote: >>>>>>> Rob Crittenden wrote: >>>>>>>> Rob Crittenden wrote: >>>>>>>>> Martin Kosek wrote: >>>>>>>>>> On 09/05/2012 08:06 PM, Rob Crittenden wrote: >>>>>>>>>>> Rob Crittenden wrote: >>>>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>>>> On 07/05/2012 08:39 PM, Rob Crittenden wrote: >>>>>>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>>>>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: >>>>>>>>>>>>>>>> Deleting a replica can leave a replication vector (RUV) on the >>>>>>>>>>>>>>>> other servers. >>>>>>>>>>>>>>>> This can confuse things if the replica is re-added, and it also >>>>>>>>>>>>>>>> causes the >>>>>>>>>>>>>>>> server to calculate changes against a server that may no longer >>>>>>>>>>>>>>>> exist. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 389-ds-base provides a new task that self-propogates itself to all >>>>>>>>>>>>>>>> available >>>>>>>>>>>>>>>> replicas to clean this RUV data. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This patch will create this task at deletion time to hopefully >>>>>>>>>>>>>>>> clean things up. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> It isn't perfect. If any replica is down or unavailable at the >>>>>>>>>>>>>>>> time >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> cleanruv task fires, and then comes back up, the old RUV data >>>>>>>>>>>>>>>> may be >>>>>>>>>>>>>>>> re-propogated around. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> To make things easier in this case I've added two new commands to >>>>>>>>>>>>>>>> ipa-replica-manage. The first lists the replication ids of all the >>>>>>>>>>>>>>>> servers we >>>>>>>>>>>>>>>> have a RUV for. Using this you can call clean_ruv with the >>>>>>>>>>>>>>>> replication id of a >>>>>>>>>>>>>>>> server that no longer exists to try the cleanallruv step again. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This is quite dangerous though. If you run cleanruv against a >>>>>>>>>>>>>>>> replica id that >>>>>>>>>>>>>>>> does exist it can cause a loss of data. I believe I've put in >>>>>>>>>>>>>>>> enough scary >>>>>>>>>>>>>>>> warnings about this. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> rob >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Good work there, this should make cleaning RUVs much easier than >>>>>>>>>>>>>>> with the >>>>>>>>>>>>>>> previous version. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is what I found during review: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1) list_ruv and clean_ruv command help in man is quite lost. I >>>>>>>>>>>>>>> think >>>>>>>>>>>>>>> it would >>>>>>>>>>>>>>> help if we for example have all info for commands indented. This >>>>>>>>>>>>>>> way >>>>>>>>>>>>>>> user could >>>>>>>>>>>>>>> simply over-look the new commands in the man page. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2) I would rename new commands to clean-ruv and list-ruv to make >>>>>>>>>>>>>>> them >>>>>>>>>>>>>>> consistent with the rest of the commands (re-initialize, >>>>>>>>>>>>>>> force-sync). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 3) It would be nice to be able to run clean_ruv command in an >>>>>>>>>>>>>>> unattended way >>>>>>>>>>>>>>> (for better testing), i.e. respect --force option as we already >>>>>>>>>>>>>>> do for >>>>>>>>>>>>>>> ipa-replica-manage del. This fix would aid test automation in the >>>>>>>>>>>>>>> future. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 4) (minor) The new question (and the del too) does not react too >>>>>>>>>>>>>>> well for >>>>>>>>>>>>>>> CTRL+D: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> # ipa-replica-manage clean_ruv 3 --force >>>>>>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>>>>>> Continue to clean? [no]: unexpected error: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 5) Help for clean_ruv command without a required parameter is quite >>>>>>>>>>>>>>> confusing >>>>>>>>>>>>>>> as it reports that command is wrong and not the parameter: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> # ipa-replica-manage clean_ruv >>>>>>>>>>>>>>> Usage: ipa-replica-manage [options] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ipa-replica-manage: error: must provide a command [clean_ruv | >>>>>>>>>>>>>>> force-sync | >>>>>>>>>>>>>>> disconnect | connect | del | re-initialize | list | list_ruv] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It seems you just forgot to specify the error message in the >>>>>>>>>>>>>>> command >>>>>>>>>>>>>>> definition >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 6) When the remote replica is down, the clean_ruv command fails >>>>>>>>>>>>>>> with an >>>>>>>>>>>>>>> unexpected error: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 >>>>>>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>>>>>> Continue to clean? [no]: y >>>>>>>>>>>>>>> unexpected error: {'desc': 'Operations error'} >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: >>>>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>>>> cleanAllRUV_task: failed >>>>>>>>>>>>>>> to connect to repl agreement connection >>>>>>>>>>>>>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> tree,cn=config), error 105 >>>>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>>>> cleanAllRUV_task: replica >>>>>>>>>>>>>>> (cn=meTovm-055.idm.lab. >>>>>>>>>>>>>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> tree, cn=config) has not been cleaned. You will need to rerun >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> CLEANALLRUV task on this replica. >>>>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>>>> cleanAllRUV_task: Task >>>>>>>>>>>>>>> failed (1) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> In this case I think we should inform user that the command failed, >>>>>>>>>>>>>>> possibly >>>>>>>>>>>>>>> because of disconnected replicas and that they could enable the >>>>>>>>>>>>>>> replicas and >>>>>>>>>>>>>>> try again. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 7) (minor) "pass" is now redundant in replication.py: >>>>>>>>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>>>>>>>>> + # We can't make the server we're removing read-only >>>>>>>>>>>>>>> but >>>>>>>>>>>>>>> + # this isn't a show-stopper >>>>>>>>>>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>>>>>>>>>> read-only, >>>>>>>>>>>>>>> continuing anyway") >>>>>>>>>>>>>>> + pass >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I think this addresses everything. >>>>>>>>>>>>>> >>>>>>>>>>>>>> rob >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, almost there! I just found one more issue which needs to be >>>>>>>>>>>>> fixed >>>>>>>>>>>>> before we push: >>>>>>>>>>>>> >>>>>>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>>>>>>>>>>> Directory Manager password: >>>>>>>>>>>>> >>>>>>>>>>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing >>>>>>>>>>>>> removal >>>>>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>>>>>>> "Can't >>>>>>>>>>>>> contact LDAP server"} >>>>>>>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>>>>>>> >>>>>>>>>>>>> There were issues removing a connection: %d format: a number is >>>>>>>>>>>>> required, not str >>>>>>>>>>>>> >>>>>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>>>>>>> "Can't >>>>>>>>>>>>> contact LDAP server"} >>>>>>>>>>>>> >>>>>>>>>>>>> This is a traceback I retrieved: >>>>>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>>>> File "/sbin/ipa-replica-manage", line 425, in del_master >>>>>>>>>>>>> del_link(realm, r, hostname, options.dirman_passwd, >>>>>>>>>>>>> force=True) >>>>>>>>>>>>> File "/sbin/ipa-replica-manage", line 271, in del_link >>>>>>>>>>>>> repl1.cleanallruv(replica_id) >>>>>>>>>>>>> File >>>>>>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>>>>>>>>>> line 1094, in cleanallruv >>>>>>>>>>>>> root_logger.debug("Creating CLEANALLRUV task for replica id >>>>>>>>>>>>> %d" % >>>>>>>>>>>>> replicaId) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> The problem here is that you don't convert replica_id to int in this >>>>>>>>>>>>> part: >>>>>>>>>>>>> + replica_id = None >>>>>>>>>>>>> + if repl2: >>>>>>>>>>>>> + replica_id = repl2._get_replica_id(repl2.conn, None) >>>>>>>>>>>>> + else: >>>>>>>>>>>>> + servers = get_ruv(realm, replica1, dirman_passwd) >>>>>>>>>>>>> + for (netloc, rid) in servers: >>>>>>>>>>>>> + if netloc.startswith(replica2): >>>>>>>>>>>>> + replica_id = rid >>>>>>>>>>>>> + break >>>>>>>>>>>>> >>>>>>>>>>>>> Martin >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Updated patch using new mechanism in 389-ds-base. This should more >>>>>>>>>>>> thoroughly clean out RUV data when a replica is being deleted, and >>>>>>>>>>>> provide for a way to delete RUV data afterwards too if necessary. >>>>>>>>>>>> >>>>>>>>>>>> rob >>>>>>>>>>> >>>>>>>>>>> Rebased patch >>>>>>>>>>> >>>>>>>>>>> rob >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 0) As I wrote in a review for your patch 1041, changelog entry slipped >>>>>>>>>> elsewhere. >>>>>>>>>> >>>>>>>>>> 1) The following KeyboardInterrupt except class looks suspicious. I >>>>>>>>>> know why >>>>>>>>>> you have it there, but since it is generally a bad thing to do, some >>>>>>>>>> comment >>>>>>>>>> why it is needed would be useful. >>>>>>>>>> >>>>>>>>>> @@ -256,6 +263,17 @@ def del_link(realm, replica1, replica2, >>>>>>>>>> dirman_passwd, >>>>>>>>>> force=False): >>>>>>>>>> repl1.delete_agreement(replica2) >>>>>>>>>> repl1.delete_referral(replica2) >>>>>>>>>> >>>>>>>>>> + if type1 == replication.IPA_REPLICA: >>>>>>>>>> + if repl2: >>>>>>>>>> + ruv = repl2._get_replica_id(repl2.conn, None) >>>>>>>>>> + else: >>>>>>>>>> + ruv = get_ruv_by_host(realm, replica1, replica2, >>>>>>>>>> dirman_passwd) >>>>>>>>>> + >>>>>>>>>> + try: >>>>>>>>>> + repl1.cleanallruv(ruv) >>>>>>>>>> + except KeyboardInterrupt: >>>>>>>>>> + pass >>>>>>>>>> + >>>>>>>>>> >>>>>>>>>> Maybe you just wanted to do some cleanup and then "raise" again? >>>>>>>>> >>>>>>>>> No, it is there because it is safe to break out of it. The task will >>>>>>>>> continue to run. I added some verbiage. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2) This is related to 1), but when some replica is down, >>>>>>>>>> "ipa-replica-manage >>>>>>>>>> del" may wait indefinitely when some remote replica is down, right? >>>>>>>>>> >>>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>>>>>> Deleting a master is irreversible. >>>>>>>>>> To reconnect to the remote master you will need to prepare a new >>>>>>>>>> replica file >>>>>>>>>> and re-install. >>>>>>>>>> Continue to delete? [no]: y >>>>>>>>>> ipa: INFO: Setting agreement >>>>>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>>>>>>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>>>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> tree,cn=config >>>>>>>>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica >>>>>>>>>> acquired >>>>>>>>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>>>>>>>> Background task created to clean replication data >>>>>>>>>> >>>>>>>>>> ... after about a minute I hit CTRL+C >>>>>>>>>> >>>>>>>>>> ^CDeleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to >>>>>>>>>> 'vm-055.idm.lab.bos.redhat.com' >>>>>>>>>> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: NS record >>>>>>>>>> does not >>>>>>>>>> contain 'vm-055.idm.lab.bos.redhat.com.' >>>>>>>>>> You may need to manually remove them from the tree >>>>>>>>>> >>>>>>>>>> I think it would be better to inform user that some remote replica is >>>>>>>>>> down or >>>>>>>>>> at least that we are waiting for the task to complete. Something like >>>>>>>>>> that: >>>>>>>>>> >>>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>>>>>> ... >>>>>>>>>> Background task created to clean replication data >>>>>>>>>> Replication data clean up may take very long time if some replica is >>>>>>>>>> unreachable >>>>>>>>>> Hit CTRL+C to interrupt the wait >>>>>>>>>> ^C Clean up wait interrupted >>>>>>>>>> .... >>>>>>>>>> [continue with del] >>>>>>>>> >>>>>>>>> Yup, did this in #1. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> 3) (minor) When there is a cleanruv task running and you run >>>>>>>>>> "ipa-replica-manage del", there is a unexpected error message with >>>>>>>>>> duplicate >>>>>>>>>> task object in LDAP: >>>>>>>>>> >>>>>>>>>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com --force >>>>>>>>>> Unable to connect to replica vm-072.idm.lab.bos.redhat.com, forcing >>>>>>>>>> removal >>>>>>>>>> FAIL >>>>>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>>>>>>> contact LDAP server"} >>>>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>>>> >>>>>>>>>> There were issues removing a connection: This entry already exists >>>>>>>>>> <<<<<<<<< >>>>>>>>>> >>>>>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>>>>>>> contact LDAP server"} >>>>>>>>>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record >>>>>>>>>> does not >>>>>>>>>> contain 'vm-072.idm.lab.bos.redhat.com.' >>>>>>>>>> You may need to manually remove them from the tree >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I think it should be enough to just catch for "entry already exists" in >>>>>>>>>> cleanallruv function, and in such case print a relevant error message >>>>>>>>>> bail out. >>>>>>>>>> Thus, self.conn.checkTask(dn, dowait=True) would not be called too. >>>>>>>>> >>>>>>>>> Good catch, fixed. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 4) (minor): In make_readonly function, there is a redundant "pass" >>>>>>>>>> statement: >>>>>>>>>> >>>>>>>>>> + def make_readonly(self): >>>>>>>>>> + """ >>>>>>>>>> + Make the current replication agreement read-only. >>>>>>>>>> + """ >>>>>>>>>> + dn = DN(('cn', 'userRoot'), ('cn', 'ldbm database'), >>>>>>>>>> + ('cn', 'plugins'), ('cn', 'config')) >>>>>>>>>> + >>>>>>>>>> + mod = [(ldap.MOD_REPLACE, 'nsslapd-readonly', 'on')] >>>>>>>>>> + try: >>>>>>>>>> + self.conn.modify_s(dn, mod) >>>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>>>> + # We can't make the server we're removing read-only but >>>>>>>>>> + # this isn't a show-stopper >>>>>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>>>>> read-only, >>>>>>>>>> continuing anyway") >>>>>>>>>> + pass <<<<<<<<<<<<<<< >>>>>>>>> >>>>>>>>> Yeah, this is one of my common mistakes. I put in a pass initially, then >>>>>>>>> add logging in front of it and forget to delete the pass. Its gone now. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 5) In clean_ruv, I think allowing a --force option to bypass the >>>>>>>>>> user_input >>>>>>>>>> would be helpful (at least for test automation): >>>>>>>>>> >>>>>>>>>> + if not ipautil.user_input("Continue to clean?", False): >>>>>>>>>> + sys.exit("Aborted") >>>>>>>>> >>>>>>>>> Yup, added. >>>>>>>>> >>>>>>>>> rob >>>>>>>> >>>>>>>> Slightly revised patch. I still had a window open with one unsaved change. >>>>>>>> >>>>>>>> rob >>>>>>>> >>>>>>> >>>>>>> Apparently there were two unsaved changes, one of which was lost. This >>>>>>> adds in >>>>>>> the 'entry already exists' fix. >>>>>>> >>>>>>> rob >>>>>>> >>>>>> >>>>>> Just one last thing (otherwise the patch is OK) - I don't think this is >>>>>> what we >>>>>> want :-) >>>>>> >>>>>> # ipa-replica-manage clean-ruv 8 >>>>>> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 >>>>>> >>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>> longer replicate so it may miss updates while the process >>>>>> is running. It would need to be re-initialized to maintain >>>>>> consistency. Be very careful. >>>>>> Continue to clean? [no]: y <<<<<< >>>>>> Aborted >>>>>> >>>>>> >>>>>> Nor this exception, (your are checking for wrong exception): >>>>>> >>>>>> # ipa-replica-manage clean-ruv 8 >>>>>> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 >>>>>> >>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>> longer replicate so it may miss updates while the process >>>>>> is running. It would need to be re-initialized to maintain >>>>>> consistency. Be very careful. >>>>>> Continue to clean? [no]: >>>>>> unexpected error: This entry already exists >>>>>> >>>>>> This is the exception: >>>>>> >>>>>> Traceback (most recent call last): >>>>>> File "/sbin/ipa-replica-manage", line 651, in >>>>>> main() >>>>>> File "/sbin/ipa-replica-manage", line 648, in main >>>>>> clean_ruv(realm, args[1], options) >>>>>> File "/sbin/ipa-replica-manage", line 373, in clean_ruv >>>>>> thisrepl.cleanallruv(ruv) >>>>>> File >>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>>> line 1136, in cleanallruv >>>>>> self.conn.addEntry(e) >>>>>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line >>>>>> 503, in >>>>>> addEntry >>>>>> self.__handle_errors(e, arg_desc=arg_desc) >>>>>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line >>>>>> 321, in >>>>>> __handle_errors >>>>>> raise errors.DuplicateEntry() >>>>>> ipalib.errors.DuplicateEntry: This entry already exists >>>>>> >>>>>> Martin >>>>>> >>>>> >>>>> Fixed that and a couple of other problems. When doing a disconnect we should >>>>> not also call clean-ruv. >>>> >>>> Ah, good self-catch. >>>> >>>>> >>>>> I also got tired of seeing crappy error messages so I added a little convert >>>>> utility. >>>>> >>>>> rob >>>> >>>> 1) There is CLEANALLRUV stuff included in 1050-3 and not here. There are also >>>> some finding for this new code. >>>> >>>> >>>> 2) We may want to bump Requires to higher version of 389-ds-base >>>> (389-ds-base-1.2.11.14-1) - it contains a fix for CLEANALLRUV+winsync bug I >>>> found earlier. >>>> >>>> >>>> 3) I just discovered another suspicious behavior. When we are deleting a >>>> master >>>> that has links also to other master(s) we delete those too. But we also >>>> automatically run CLEANALLRUV in these cases, so we may end up in multiple >>>> tasks being started on different masters - this does not look right. >>>> >>>> I think we may rather want to at first delete all links and then run >>>> CLEANALLRUV task, just for one time. This is what I get with current code: >>>> >>>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com >>>> Directory Manager password: >>>> >>>> Deleting a master is irreversible. >>>> To reconnect to the remote master you will need to prepare a new replica file >>>> and re-install. >>>> Continue to delete? [no]: yes >>>> ipa: INFO: Setting agreement >>>> cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>> >>>> >>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>> cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>> >>>> >>>> tree,cn=config >>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica acquired >>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>> Background task created to clean replication data. This may take a while. >>>> This may be safely interrupted with Ctrl+C >>>> >>>> ^CWait for task interrupted. It will continue to run in the background >>>> >>>> Deleted replication agreement from 'vm-055.idm.lab.bos.redhat.com' to >>>> 'vm-072.idm.lab.bos.redhat.com' >>>> ipa: INFO: Setting agreement >>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>> >>>> >>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>> >>>> >>>> tree,cn=config >>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica acquired >>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>> Background task created to clean replication data. This may take a while. >>>> This may be safely interrupted with Ctrl+C >>>> >>>> ^CWait for task interrupted. It will continue to run in the background >>>> >>>> Deleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to >>>> 'vm-072.idm.lab.bos.redhat.com' >>>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record does >>>> not >>>> contain 'vm-072.idm.lab.bos.redhat.com.' >>>> You may need to manually remove them from the tree >>>> >>>> Martin >>>> >>> >>> All issues addressed and I pulled in abort-clean-ruv from 1050. I added a >>> list-clean-ruv command as well. >>> >>> rob >> >> 1) Patch 1031-9 needs to get squashed with 1031-8 >> >> >> 2) Patch needs a rebase (conflict in freeipa.spec.in) >> >> >> 3) New list-clean-ruv man entry is not right: >> >> list-clean-ruv [REPLICATION_ID] >> - List all running CLEANALLRUV and abort CLEANALLRUV tasks. >> >> REPLICATION_ID is not its argument. > > Fixed 1-3. > >> Btw. new list-clean-ruv command proved very useful for me. >> >> 4) I just found out we need to do a better job with make_readonly() command. I >> get into trouble when disconnecting one link to a remote replica as it was >> marked readonly and then I was then unable to manage the disconnected replica >> properly (vm-072 is the replica made readonly): > > Ok, I reset read-only after we delete the agreements. That fixed things up for > me. I disconnected a replica and was able to modify entries on that replica > afterwards. > > This affected the --cleanup command too, it would otherwise have succeeded I > think. > > I tested with an A - B - C - A agreement loop. I disconnected A and C and > confirmed I could still update entries on C. Then I deleted C, then B, and made > sure output looked right, I could still manage entries, etc. > > rob > >> >> [root at vm-055 ~]# ipa-replica-manage disconnect vm-072.idm.lab.bos.redhat.com >> >> [root at vm-072 ~]# ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >> Deleting a master is irreversible. >> To reconnect to the remote master you will need to prepare a new replica file >> and re-install. >> Continue to delete? [no]: yes >> Deleting replication agreements between vm-055.idm.lab.bos.redhat.com and >> vm-072.idm.lab.bos.redhat.com >> ipa: INFO: Setting agreement >> cn=meTovm-072.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >> >> tree,cn=config schedule to 2358-2359 0 to force synch >> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >> cn=meTovm-072.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >> >> tree,cn=config >> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica acquired >> successfully: Incremental update succeeded: start: 0: end: 0 >> Deleted replication agreement from 'vm-072.idm.lab.bos.redhat.com' to >> 'vm-055.idm.lab.bos.redhat.com' >> Unable to remove replication agreement for vm-055.idm.lab.bos.redhat.com from >> vm-072.idm.lab.bos.redhat.com. >> Background task created to clean replication data. This may take a while. >> This may be safely interrupted with Ctrl+C >> ^CWait for task interrupted. It will continue to run in the background >> >> Failed to cleanup vm-055.idm.lab.bos.redhat.com entries: Server is unwilling to >> perform: database is read-only arguments: >> dn=krbprincipalname=ldap/vm-055.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM,cn=services,cn=accounts,dc=idm,dc=lab,dc=bos,dc=redhat,dc=com >> >> >> You may need to manually remove them from the tree >> ipa: INFO: Unhandled LDAPError: {'info': 'database is read-only', 'desc': >> 'Server is unwilling to perform'} >> >> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: Server is >> unwilling to perform: database is read-only >> >> You may need to manually remove them from the tree >> >> >> --cleanup did not work for me as well: >> [root at vm-072 ~]# ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >> --cleanup >> Cleaning a master is irreversible. >> This should not normally be require, so use cautiously. >> Continue to clean master? [no]: yes >> unexpected error: Server is unwilling to perform: database is read-only >> arguments: >> dn=krbprincipalname=ldap/vm-055.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM,cn=services,cn=accounts,dc=idm,dc=lab,dc=bos,dc=redhat,dc=com >> >> >> Martin >> > I think you sent a wrong patch... Martin From rcritten at redhat.com Mon Sep 17 14:15:30 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 17 Sep 2012 10:15:30 -0400 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <50572F9A.3040605@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> <50479491.1060200@redhat.com> <50488DD0.1080507@redhat.com> <5048BEF8.5000401@redhat.com> <5048C46A.8020207@redhat.com> <5048C754.2030708@redhat.com> <5048C9BE.1050005@redhat.com> <504912F9.1020907@redhat.com> <50531DDC.5050007@redhat.com> <505382CF.1030901@redhat.com> <5056E754.1080201@redhat.com> <50572DEF.7020104@redhat.com> <50572F9A.3040605@redhat.com> Message-ID: <50573082.3020601@redhat.com> Martin Kosek wrote: > On 09/17/2012 04:04 PM, Rob Crittenden wrote: >> Martin Kosek wrote: >>> On 09/14/2012 09:17 PM, Rob Crittenden wrote: >>>> Martin Kosek wrote: >>>>> On 09/06/2012 11:17 PM, Rob Crittenden wrote: >>>>>> Martin Kosek wrote: >>>>>>> On 09/06/2012 05:55 PM, Rob Crittenden wrote: >>>>>>>> Rob Crittenden wrote: >>>>>>>>> Rob Crittenden wrote: >>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>> On 09/05/2012 08:06 PM, Rob Crittenden wrote: >>>>>>>>>>>> Rob Crittenden wrote: >>>>>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>>>>> On 07/05/2012 08:39 PM, Rob Crittenden wrote: >>>>>>>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>>>>>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: >>>>>>>>>>>>>>>>> Deleting a replica can leave a replication vector (RUV) on the >>>>>>>>>>>>>>>>> other servers. >>>>>>>>>>>>>>>>> This can confuse things if the replica is re-added, and it also >>>>>>>>>>>>>>>>> causes the >>>>>>>>>>>>>>>>> server to calculate changes against a server that may no longer >>>>>>>>>>>>>>>>> exist. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 389-ds-base provides a new task that self-propogates itself to all >>>>>>>>>>>>>>>>> available >>>>>>>>>>>>>>>>> replicas to clean this RUV data. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> This patch will create this task at deletion time to hopefully >>>>>>>>>>>>>>>>> clean things up. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> It isn't perfect. If any replica is down or unavailable at the >>>>>>>>>>>>>>>>> time >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> cleanruv task fires, and then comes back up, the old RUV data >>>>>>>>>>>>>>>>> may be >>>>>>>>>>>>>>>>> re-propogated around. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> To make things easier in this case I've added two new commands to >>>>>>>>>>>>>>>>> ipa-replica-manage. The first lists the replication ids of all the >>>>>>>>>>>>>>>>> servers we >>>>>>>>>>>>>>>>> have a RUV for. Using this you can call clean_ruv with the >>>>>>>>>>>>>>>>> replication id of a >>>>>>>>>>>>>>>>> server that no longer exists to try the cleanallruv step again. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> This is quite dangerous though. If you run cleanruv against a >>>>>>>>>>>>>>>>> replica id that >>>>>>>>>>>>>>>>> does exist it can cause a loss of data. I believe I've put in >>>>>>>>>>>>>>>>> enough scary >>>>>>>>>>>>>>>>> warnings about this. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> rob >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Good work there, this should make cleaning RUVs much easier than >>>>>>>>>>>>>>>> with the >>>>>>>>>>>>>>>> previous version. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This is what I found during review: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1) list_ruv and clean_ruv command help in man is quite lost. I >>>>>>>>>>>>>>>> think >>>>>>>>>>>>>>>> it would >>>>>>>>>>>>>>>> help if we for example have all info for commands indented. This >>>>>>>>>>>>>>>> way >>>>>>>>>>>>>>>> user could >>>>>>>>>>>>>>>> simply over-look the new commands in the man page. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2) I would rename new commands to clean-ruv and list-ruv to make >>>>>>>>>>>>>>>> them >>>>>>>>>>>>>>>> consistent with the rest of the commands (re-initialize, >>>>>>>>>>>>>>>> force-sync). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 3) It would be nice to be able to run clean_ruv command in an >>>>>>>>>>>>>>>> unattended way >>>>>>>>>>>>>>>> (for better testing), i.e. respect --force option as we already >>>>>>>>>>>>>>>> do for >>>>>>>>>>>>>>>> ipa-replica-manage del. This fix would aid test automation in the >>>>>>>>>>>>>>>> future. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 4) (minor) The new question (and the del too) does not react too >>>>>>>>>>>>>>>> well for >>>>>>>>>>>>>>>> CTRL+D: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> # ipa-replica-manage clean_ruv 3 --force >>>>>>>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>>>>>>> Continue to clean? [no]: unexpected error: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 5) Help for clean_ruv command without a required parameter is quite >>>>>>>>>>>>>>>> confusing >>>>>>>>>>>>>>>> as it reports that command is wrong and not the parameter: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> # ipa-replica-manage clean_ruv >>>>>>>>>>>>>>>> Usage: ipa-replica-manage [options] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ipa-replica-manage: error: must provide a command [clean_ruv | >>>>>>>>>>>>>>>> force-sync | >>>>>>>>>>>>>>>> disconnect | connect | del | re-initialize | list | list_ruv] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> It seems you just forgot to specify the error message in the >>>>>>>>>>>>>>>> command >>>>>>>>>>>>>>>> definition >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 6) When the remote replica is down, the clean_ruv command fails >>>>>>>>>>>>>>>> with an >>>>>>>>>>>>>>>> unexpected error: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 >>>>>>>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>>>>>>> Continue to clean? [no]: y >>>>>>>>>>>>>>>> unexpected error: {'desc': 'Operations error'} >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: >>>>>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>>>>> cleanAllRUV_task: failed >>>>>>>>>>>>>>>> to connect to repl agreement connection >>>>>>>>>>>>>>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> tree,cn=config), error 105 >>>>>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>>>>> cleanAllRUV_task: replica >>>>>>>>>>>>>>>> (cn=meTovm-055.idm.lab. >>>>>>>>>>>>>>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> tree, cn=config) has not been cleaned. You will need to rerun >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> CLEANALLRUV task on this replica. >>>>>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>>>>> cleanAllRUV_task: Task >>>>>>>>>>>>>>>> failed (1) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> In this case I think we should inform user that the command failed, >>>>>>>>>>>>>>>> possibly >>>>>>>>>>>>>>>> because of disconnected replicas and that they could enable the >>>>>>>>>>>>>>>> replicas and >>>>>>>>>>>>>>>> try again. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 7) (minor) "pass" is now redundant in replication.py: >>>>>>>>>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>>>>>>>>>> + # We can't make the server we're removing read-only >>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>> + # this isn't a show-stopper >>>>>>>>>>>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>>>>>>>>>>> read-only, >>>>>>>>>>>>>>>> continuing anyway") >>>>>>>>>>>>>>>> + pass >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I think this addresses everything. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> rob >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, almost there! I just found one more issue which needs to be >>>>>>>>>>>>>> fixed >>>>>>>>>>>>>> before we push: >>>>>>>>>>>>>> >>>>>>>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>>>>>>>>>>>> Directory Manager password: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing >>>>>>>>>>>>>> removal >>>>>>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>>>>>>>> "Can't >>>>>>>>>>>>>> contact LDAP server"} >>>>>>>>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>>>>>>>> >>>>>>>>>>>>>> There were issues removing a connection: %d format: a number is >>>>>>>>>>>>>> required, not str >>>>>>>>>>>>>> >>>>>>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>>>>>>>> "Can't >>>>>>>>>>>>>> contact LDAP server"} >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is a traceback I retrieved: >>>>>>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>>>>> File "/sbin/ipa-replica-manage", line 425, in del_master >>>>>>>>>>>>>> del_link(realm, r, hostname, options.dirman_passwd, >>>>>>>>>>>>>> force=True) >>>>>>>>>>>>>> File "/sbin/ipa-replica-manage", line 271, in del_link >>>>>>>>>>>>>> repl1.cleanallruv(replica_id) >>>>>>>>>>>>>> File >>>>>>>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>>>>>>>>>>> line 1094, in cleanallruv >>>>>>>>>>>>>> root_logger.debug("Creating CLEANALLRUV task for replica id >>>>>>>>>>>>>> %d" % >>>>>>>>>>>>>> replicaId) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> The problem here is that you don't convert replica_id to int in this >>>>>>>>>>>>>> part: >>>>>>>>>>>>>> + replica_id = None >>>>>>>>>>>>>> + if repl2: >>>>>>>>>>>>>> + replica_id = repl2._get_replica_id(repl2.conn, None) >>>>>>>>>>>>>> + else: >>>>>>>>>>>>>> + servers = get_ruv(realm, replica1, dirman_passwd) >>>>>>>>>>>>>> + for (netloc, rid) in servers: >>>>>>>>>>>>>> + if netloc.startswith(replica2): >>>>>>>>>>>>>> + replica_id = rid >>>>>>>>>>>>>> + break >>>>>>>>>>>>>> >>>>>>>>>>>>>> Martin >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Updated patch using new mechanism in 389-ds-base. This should more >>>>>>>>>>>>> thoroughly clean out RUV data when a replica is being deleted, and >>>>>>>>>>>>> provide for a way to delete RUV data afterwards too if necessary. >>>>>>>>>>>>> >>>>>>>>>>>>> rob >>>>>>>>>>>> >>>>>>>>>>>> Rebased patch >>>>>>>>>>>> >>>>>>>>>>>> rob >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 0) As I wrote in a review for your patch 1041, changelog entry slipped >>>>>>>>>>> elsewhere. >>>>>>>>>>> >>>>>>>>>>> 1) The following KeyboardInterrupt except class looks suspicious. I >>>>>>>>>>> know why >>>>>>>>>>> you have it there, but since it is generally a bad thing to do, some >>>>>>>>>>> comment >>>>>>>>>>> why it is needed would be useful. >>>>>>>>>>> >>>>>>>>>>> @@ -256,6 +263,17 @@ def del_link(realm, replica1, replica2, >>>>>>>>>>> dirman_passwd, >>>>>>>>>>> force=False): >>>>>>>>>>> repl1.delete_agreement(replica2) >>>>>>>>>>> repl1.delete_referral(replica2) >>>>>>>>>>> >>>>>>>>>>> + if type1 == replication.IPA_REPLICA: >>>>>>>>>>> + if repl2: >>>>>>>>>>> + ruv = repl2._get_replica_id(repl2.conn, None) >>>>>>>>>>> + else: >>>>>>>>>>> + ruv = get_ruv_by_host(realm, replica1, replica2, >>>>>>>>>>> dirman_passwd) >>>>>>>>>>> + >>>>>>>>>>> + try: >>>>>>>>>>> + repl1.cleanallruv(ruv) >>>>>>>>>>> + except KeyboardInterrupt: >>>>>>>>>>> + pass >>>>>>>>>>> + >>>>>>>>>>> >>>>>>>>>>> Maybe you just wanted to do some cleanup and then "raise" again? >>>>>>>>>> >>>>>>>>>> No, it is there because it is safe to break out of it. The task will >>>>>>>>>> continue to run. I added some verbiage. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2) This is related to 1), but when some replica is down, >>>>>>>>>>> "ipa-replica-manage >>>>>>>>>>> del" may wait indefinitely when some remote replica is down, right? >>>>>>>>>>> >>>>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>>>>>>> Deleting a master is irreversible. >>>>>>>>>>> To reconnect to the remote master you will need to prepare a new >>>>>>>>>>> replica file >>>>>>>>>>> and re-install. >>>>>>>>>>> Continue to delete? [no]: y >>>>>>>>>>> ipa: INFO: Setting agreement >>>>>>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>>>>>>>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>>>>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> tree,cn=config >>>>>>>>>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica >>>>>>>>>>> acquired >>>>>>>>>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>>>>>>>>> Background task created to clean replication data >>>>>>>>>>> >>>>>>>>>>> ... after about a minute I hit CTRL+C >>>>>>>>>>> >>>>>>>>>>> ^CDeleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to >>>>>>>>>>> 'vm-055.idm.lab.bos.redhat.com' >>>>>>>>>>> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: NS record >>>>>>>>>>> does not >>>>>>>>>>> contain 'vm-055.idm.lab.bos.redhat.com.' >>>>>>>>>>> You may need to manually remove them from the tree >>>>>>>>>>> >>>>>>>>>>> I think it would be better to inform user that some remote replica is >>>>>>>>>>> down or >>>>>>>>>>> at least that we are waiting for the task to complete. Something like >>>>>>>>>>> that: >>>>>>>>>>> >>>>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>>>>>>> ... >>>>>>>>>>> Background task created to clean replication data >>>>>>>>>>> Replication data clean up may take very long time if some replica is >>>>>>>>>>> unreachable >>>>>>>>>>> Hit CTRL+C to interrupt the wait >>>>>>>>>>> ^C Clean up wait interrupted >>>>>>>>>>> .... >>>>>>>>>>> [continue with del] >>>>>>>>>> >>>>>>>>>> Yup, did this in #1. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 3) (minor) When there is a cleanruv task running and you run >>>>>>>>>>> "ipa-replica-manage del", there is a unexpected error message with >>>>>>>>>>> duplicate >>>>>>>>>>> task object in LDAP: >>>>>>>>>>> >>>>>>>>>>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com --force >>>>>>>>>>> Unable to connect to replica vm-072.idm.lab.bos.redhat.com, forcing >>>>>>>>>>> removal >>>>>>>>>>> FAIL >>>>>>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>>>>>>>> contact LDAP server"} >>>>>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>>>>> >>>>>>>>>>> There were issues removing a connection: This entry already exists >>>>>>>>>>> <<<<<<<<< >>>>>>>>>>> >>>>>>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': "Can't >>>>>>>>>>> contact LDAP server"} >>>>>>>>>>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record >>>>>>>>>>> does not >>>>>>>>>>> contain 'vm-072.idm.lab.bos.redhat.com.' >>>>>>>>>>> You may need to manually remove them from the tree >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I think it should be enough to just catch for "entry already exists" in >>>>>>>>>>> cleanallruv function, and in such case print a relevant error message >>>>>>>>>>> bail out. >>>>>>>>>>> Thus, self.conn.checkTask(dn, dowait=True) would not be called too. >>>>>>>>>> >>>>>>>>>> Good catch, fixed. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 4) (minor): In make_readonly function, there is a redundant "pass" >>>>>>>>>>> statement: >>>>>>>>>>> >>>>>>>>>>> + def make_readonly(self): >>>>>>>>>>> + """ >>>>>>>>>>> + Make the current replication agreement read-only. >>>>>>>>>>> + """ >>>>>>>>>>> + dn = DN(('cn', 'userRoot'), ('cn', 'ldbm database'), >>>>>>>>>>> + ('cn', 'plugins'), ('cn', 'config')) >>>>>>>>>>> + >>>>>>>>>>> + mod = [(ldap.MOD_REPLACE, 'nsslapd-readonly', 'on')] >>>>>>>>>>> + try: >>>>>>>>>>> + self.conn.modify_s(dn, mod) >>>>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>>>>> + # We can't make the server we're removing read-only but >>>>>>>>>>> + # this isn't a show-stopper >>>>>>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>>>>>> read-only, >>>>>>>>>>> continuing anyway") >>>>>>>>>>> + pass <<<<<<<<<<<<<<< >>>>>>>>>> >>>>>>>>>> Yeah, this is one of my common mistakes. I put in a pass initially, then >>>>>>>>>> add logging in front of it and forget to delete the pass. Its gone now. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 5) In clean_ruv, I think allowing a --force option to bypass the >>>>>>>>>>> user_input >>>>>>>>>>> would be helpful (at least for test automation): >>>>>>>>>>> >>>>>>>>>>> + if not ipautil.user_input("Continue to clean?", False): >>>>>>>>>>> + sys.exit("Aborted") >>>>>>>>>> >>>>>>>>>> Yup, added. >>>>>>>>>> >>>>>>>>>> rob >>>>>>>>> >>>>>>>>> Slightly revised patch. I still had a window open with one unsaved change. >>>>>>>>> >>>>>>>>> rob >>>>>>>>> >>>>>>>> >>>>>>>> Apparently there were two unsaved changes, one of which was lost. This >>>>>>>> adds in >>>>>>>> the 'entry already exists' fix. >>>>>>>> >>>>>>>> rob >>>>>>>> >>>>>>> >>>>>>> Just one last thing (otherwise the patch is OK) - I don't think this is >>>>>>> what we >>>>>>> want :-) >>>>>>> >>>>>>> # ipa-replica-manage clean-ruv 8 >>>>>>> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 >>>>>>> >>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>> longer replicate so it may miss updates while the process >>>>>>> is running. It would need to be re-initialized to maintain >>>>>>> consistency. Be very careful. >>>>>>> Continue to clean? [no]: y <<<<<< >>>>>>> Aborted >>>>>>> >>>>>>> >>>>>>> Nor this exception, (your are checking for wrong exception): >>>>>>> >>>>>>> # ipa-replica-manage clean-ruv 8 >>>>>>> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 >>>>>>> >>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>> longer replicate so it may miss updates while the process >>>>>>> is running. It would need to be re-initialized to maintain >>>>>>> consistency. Be very careful. >>>>>>> Continue to clean? [no]: >>>>>>> unexpected error: This entry already exists >>>>>>> >>>>>>> This is the exception: >>>>>>> >>>>>>> Traceback (most recent call last): >>>>>>> File "/sbin/ipa-replica-manage", line 651, in >>>>>>> main() >>>>>>> File "/sbin/ipa-replica-manage", line 648, in main >>>>>>> clean_ruv(realm, args[1], options) >>>>>>> File "/sbin/ipa-replica-manage", line 373, in clean_ruv >>>>>>> thisrepl.cleanallruv(ruv) >>>>>>> File >>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>>>> line 1136, in cleanallruv >>>>>>> self.conn.addEntry(e) >>>>>>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line >>>>>>> 503, in >>>>>>> addEntry >>>>>>> self.__handle_errors(e, arg_desc=arg_desc) >>>>>>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line >>>>>>> 321, in >>>>>>> __handle_errors >>>>>>> raise errors.DuplicateEntry() >>>>>>> ipalib.errors.DuplicateEntry: This entry already exists >>>>>>> >>>>>>> Martin >>>>>>> >>>>>> >>>>>> Fixed that and a couple of other problems. When doing a disconnect we should >>>>>> not also call clean-ruv. >>>>> >>>>> Ah, good self-catch. >>>>> >>>>>> >>>>>> I also got tired of seeing crappy error messages so I added a little convert >>>>>> utility. >>>>>> >>>>>> rob >>>>> >>>>> 1) There is CLEANALLRUV stuff included in 1050-3 and not here. There are also >>>>> some finding for this new code. >>>>> >>>>> >>>>> 2) We may want to bump Requires to higher version of 389-ds-base >>>>> (389-ds-base-1.2.11.14-1) - it contains a fix for CLEANALLRUV+winsync bug I >>>>> found earlier. >>>>> >>>>> >>>>> 3) I just discovered another suspicious behavior. When we are deleting a >>>>> master >>>>> that has links also to other master(s) we delete those too. But we also >>>>> automatically run CLEANALLRUV in these cases, so we may end up in multiple >>>>> tasks being started on different masters - this does not look right. >>>>> >>>>> I think we may rather want to at first delete all links and then run >>>>> CLEANALLRUV task, just for one time. This is what I get with current code: >>>>> >>>>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com >>>>> Directory Manager password: >>>>> >>>>> Deleting a master is irreversible. >>>>> To reconnect to the remote master you will need to prepare a new replica file >>>>> and re-install. >>>>> Continue to delete? [no]: yes >>>>> ipa: INFO: Setting agreement >>>>> cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>> >>>>> >>>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>>> cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>> >>>>> >>>>> tree,cn=config >>>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica acquired >>>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>>> Background task created to clean replication data. This may take a while. >>>>> This may be safely interrupted with Ctrl+C >>>>> >>>>> ^CWait for task interrupted. It will continue to run in the background >>>>> >>>>> Deleted replication agreement from 'vm-055.idm.lab.bos.redhat.com' to >>>>> 'vm-072.idm.lab.bos.redhat.com' >>>>> ipa: INFO: Setting agreement >>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>> >>>>> >>>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>> >>>>> >>>>> tree,cn=config >>>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica acquired >>>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>>> Background task created to clean replication data. This may take a while. >>>>> This may be safely interrupted with Ctrl+C >>>>> >>>>> ^CWait for task interrupted. It will continue to run in the background >>>>> >>>>> Deleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to >>>>> 'vm-072.idm.lab.bos.redhat.com' >>>>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record does >>>>> not >>>>> contain 'vm-072.idm.lab.bos.redhat.com.' >>>>> You may need to manually remove them from the tree >>>>> >>>>> Martin >>>>> >>>> >>>> All issues addressed and I pulled in abort-clean-ruv from 1050. I added a >>>> list-clean-ruv command as well. >>>> >>>> rob >>> >>> 1) Patch 1031-9 needs to get squashed with 1031-8 >>> >>> >>> 2) Patch needs a rebase (conflict in freeipa.spec.in) >>> >>> >>> 3) New list-clean-ruv man entry is not right: >>> >>> list-clean-ruv [REPLICATION_ID] >>> - List all running CLEANALLRUV and abort CLEANALLRUV tasks. >>> >>> REPLICATION_ID is not its argument. >> >> Fixed 1-3. >> >>> Btw. new list-clean-ruv command proved very useful for me. >>> >>> 4) I just found out we need to do a better job with make_readonly() command. I >>> get into trouble when disconnecting one link to a remote replica as it was >>> marked readonly and then I was then unable to manage the disconnected replica >>> properly (vm-072 is the replica made readonly): >> >> Ok, I reset read-only after we delete the agreements. That fixed things up for >> me. I disconnected a replica and was able to modify entries on that replica >> afterwards. >> >> This affected the --cleanup command too, it would otherwise have succeeded I >> think. >> >> I tested with an A - B - C - A agreement loop. I disconnected A and C and >> confirmed I could still update entries on C. Then I deleted C, then B, and made >> sure output looked right, I could still manage entries, etc. >> >> rob >> >>> >>> [root at vm-055 ~]# ipa-replica-manage disconnect vm-072.idm.lab.bos.redhat.com >>> >>> [root at vm-072 ~]# ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>> Deleting a master is irreversible. >>> To reconnect to the remote master you will need to prepare a new replica file >>> and re-install. >>> Continue to delete? [no]: yes >>> Deleting replication agreements between vm-055.idm.lab.bos.redhat.com and >>> vm-072.idm.lab.bos.redhat.com >>> ipa: INFO: Setting agreement >>> cn=meTovm-072.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>> >>> tree,cn=config schedule to 2358-2359 0 to force synch >>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>> cn=meTovm-072.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>> >>> tree,cn=config >>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica acquired >>> successfully: Incremental update succeeded: start: 0: end: 0 >>> Deleted replication agreement from 'vm-072.idm.lab.bos.redhat.com' to >>> 'vm-055.idm.lab.bos.redhat.com' >>> Unable to remove replication agreement for vm-055.idm.lab.bos.redhat.com from >>> vm-072.idm.lab.bos.redhat.com. >>> Background task created to clean replication data. This may take a while. >>> This may be safely interrupted with Ctrl+C >>> ^CWait for task interrupted. It will continue to run in the background >>> >>> Failed to cleanup vm-055.idm.lab.bos.redhat.com entries: Server is unwilling to >>> perform: database is read-only arguments: >>> dn=krbprincipalname=ldap/vm-055.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM,cn=services,cn=accounts,dc=idm,dc=lab,dc=bos,dc=redhat,dc=com >>> >>> >>> You may need to manually remove them from the tree >>> ipa: INFO: Unhandled LDAPError: {'info': 'database is read-only', 'desc': >>> 'Server is unwilling to perform'} >>> >>> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: Server is >>> unwilling to perform: database is read-only >>> >>> You may need to manually remove them from the tree >>> >>> >>> --cleanup did not work for me as well: >>> [root at vm-072 ~]# ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>> --cleanup >>> Cleaning a master is irreversible. >>> This should not normally be require, so use cautiously. >>> Continue to clean master? [no]: yes >>> unexpected error: Server is unwilling to perform: database is read-only >>> arguments: >>> dn=krbprincipalname=ldap/vm-055.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM,cn=services,cn=accounts,dc=idm,dc=lab,dc=bos,dc=redhat,dc=com >>> >>> >>> Martin >>> >> > > I think you sent a wrong patch... > > Martin > I hate Mondays. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-rcrit-1031-11-cleanruv.patch Type: text/x-diff Size: 27405 bytes Desc: not available URL: From alee at redhat.com Mon Sep 17 14:25:22 2012 From: alee at redhat.com (Ade Lee) Date: Mon, 17 Sep 2012 10:25:22 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <50571879.3080401@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> <1347329065.24731.101.camel@aleeredhat.laptop> <504F318F.3030603@redhat.com> <504F35AD.3030905@redhat.com> <1347373691.24731.125.camel@aleeredhat.laptop> <504F4CE0.9080706@redhat.com> <504F67C5.4000600@redhat.com> <504F86C0.6060907@redhat.com> <5050BB14.3050502@redhat.com> <5053350D.5010805@redhat.com> <50539F44.5040308@redhat.com> <50571879.3080401@redhat.com> Message-ID: <1347891922.24731.157.camel@aleeredhat.laptop> On Mon, 2012-09-17 at 14:32 +0200, Petr Viktorin wrote: > On 09/14/2012 11:19 PM, Rob Crittenden wrote: > > Petr Viktorin wrote: > >> On 09/12/2012 06:40 PM, Petr Viktorin wrote: > >>> A new Dogtag build with changed pkispawn/pkidestroy locations should be > >>> out later today. The attached patch should work with that build. > > > > Fresh install is failing in F-18. > > > > ki-tools-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.i686 > > pki-base-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.noarch > > pki-server-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.noarch > > pki-silent-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.noarch > > pki-symkey-9.0.21-1.fc18.x86_64 > > dogtag-pki-ca-theme-10.0.0-0.1.a1.20120914T0604zgit69c0684.fc18.noarch > > pki-selinux-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.noarch > > pki-ca-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.noarch > > pki-setup-9.0.21-1.fc18.noarch > > > > > > rob > > > > > > Ade, your patch adds a step of moving > /var/lib/pki/pki-tomcat/alias/ca_admin_cert.p12 to /root/ca-agent.p12 > right after calling pkispawn. > It seems the file is not created on f18. Did something change in Dogtag > or are we calling it incorrectly? > The failure of that step often indicates a failure of the previous configure() step. That is - moving that file fails because it was not created, because configuration fails. Rob's logs seem to indicate some kind of classpath issue with the jython code in pkispawn which calls configure() on the server. I set up an f18 machine and was able to configure an instance (outside of IPA) Will now try with the ipa code (and your patches) to see if I can reproduce. Ade > From pspacek at redhat.com Mon Sep 17 15:07:44 2012 From: pspacek at redhat.com (Petr Spacek) Date: Mon, 17 Sep 2012 17:07:44 +0200 Subject: [Freeipa-devel] [PATCH 0064] Improve log message about improperly formated Resource Records Message-ID: <50573CC0.8050308@redhat.com> Hello, this patch adds DN to log message about improperly formated Resource Records. Petr^2 Spacek -------------- next part -------------- A non-text attachment was scrubbed... Name: bind-dyndb-ldap-pspacek-0064-Improve-log-message-about-improperly-formated-Resour.patch Type: text/x-patch Size: 2503 bytes Desc: not available URL: From rcritten at redhat.com Mon Sep 17 15:09:54 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 17 Sep 2012 11:09:54 -0400 Subject: [Freeipa-devel] [PATCH] 309 Fix addattr internal error In-Reply-To: <5052D160.1000701@redhat.com> References: <5051EEB8.6050303@redhat.com> <505231C6.9060105@redhat.com> <5052D160.1000701@redhat.com> Message-ID: <50573D42.10808@redhat.com> Martin Kosek wrote: > On 09/13/2012 09:19 PM, Rob Crittenden wrote: >> Martin Kosek wrote: >>> When ADD command is being executed and a single-value object attribute >>> is being set with both option and addattr IPA ends up in an internal >>> error. >>> >>> Make better value sanitizing job in this case and let IPA throw >>> a user-friendly error. Unit test exercising this situation is added. >>> >>> https://fedorahosted.org/freeipa/ticket/2429 >> >> + if not isinstance(val, (list, tuple)): >> + val = [val] >> + val.extend(adddict[attr]) >> >> I val is a tuple the extend is still going to fail. Can val ever be a tuple? If >> so we'd need to convert it to a list. >> >> rob > > I don't think it could be a tuple, I am about 99% certain. So for this 1% I > added a special clause for tuple. Patch attached. > > We will be able to be even more certain when Honza finishes his strict encoding > patch he was working on in the summer. With his patch, the attributes should > always be a list. > > Martin > ACK, pushed to master and ipa-3-0 rob From rcritten at redhat.com Mon Sep 17 15:17:23 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 17 Sep 2012 11:17:23 -0400 Subject: [Freeipa-devel] [PATCH] 305-308 Expand Referential Integrity checks In-Reply-To: <5052ED9B.7000609@redhat.com> References: <50507B31.6070804@redhat.com> <50520C70.7080800@redhat.com> <5052ED9B.7000609@redhat.com> Message-ID: <50573F03.80000@redhat.com> Martin Kosek wrote: > On 09/13/2012 06:40 PM, Rob Crittenden wrote: >> Martin Kosek wrote: >>> To test, add sudo commands, hosts or users to a sudo rule or hbac rule and then >>> rename or delete the linked object. After the update, the links should be >>> amended. >>> >>> --------- >>> >>> Many attributes in IPA (e.g. manager, memberuser, managedby, ...) >>> are used to store DNs of linked objects in IPA (users, hosts, sudo >>> commands, etc.). However, when the linked objects is deleted or >>> renamed, the attribute pointing to it stays with the objects and >>> thus may create a dangling link causing issues in client software >>> reading the data. >>> >>> Directory Server has a plugin to enforce referential integrity (RI) >>> by checking DEL and MODRDN operations and updating affected links. >>> It was already used for manager and secretary attributes and >>> should be expanded for the missing attributes to avoid dangling >>> links. >>> >>> As a prerequisite, all attributes checked for RI must have pres >>> and eq indexes to avoid performance issues. The following indexes >>> have been added: >>> * manager (pres index only) >>> * secretary (pres index only) >>> * memberHost >>> * memberUser >>> * sourcehost >>> * memberservice >>> * managedby >>> * memberallowcmd >>> * memberdenycmd >>> * ipasudorunas >>> * ipasudorunasgroup >>> >>> Referential Integrity plugin was updated to check all these >>> attributes. >>> >>> Note: this update will only fix RI on one master as RI plugin does >>> not check replicated operations. >>> >>> https://fedorahosted.org/freeipa/ticket/2866 >> >> These patches look good but I'd like to see some tests associated with the >> referential integrity changes in patch 308. I'm not sure we need a test for >> every single combination where RI comes into play but at least testing that the >> original sequence (sudorule/sudocmd) works as expected. >> >> rob > > Right, I should have seen that coming. I want this feature to be checked > properly so I added a tests for all RI-checked attributes. > > Patches attached. > > Martin > ACK, pushed to master and ipa-3-0 rob From alee at redhat.com Mon Sep 17 15:39:45 2012 From: alee at redhat.com (Ade Lee) Date: Mon, 17 Sep 2012 11:39:45 -0400 Subject: [Freeipa-devel] [PATCH] Patch to allow IPA to work with dogtag 10 on f18 In-Reply-To: <50571879.3080401@redhat.com> References: <1344456327.21349.37.camel@aleeredhat.laptop> <502BB362.4090300@redhat.com> <1345096405.21349.176.camel@aleeredhat.laptop> <502C9D57.2060105@redhat.com> <1345116527.21349.178.camel@aleeredhat.laptop> <1345135508.2452.4.camel@priserak> <1345210494.21349.190.camel@aleeredhat.laptop> <1345219494.21349.193.camel@aleeredhat.laptop> <503CCA45.1010807@redhat.com> <5040CFEF.8040100@redhat.com> <5044A40B.6070604@redhat.com> <5047B3F8.4060406@redhat.com> <1346893277.24731.60.camel@aleeredhat.laptop> <50489897.2090601@redhat.com> <50489B48.70503@redhat.com> <504DF549.5040007@redhat.com> <504E1E45.5090608@redhat.com> <504E5480.3000201@redhat.com> <1347329065.24731.101.camel@aleeredhat.laptop> <504F318F.3030603@redhat.com> <504F35AD.3030905@redhat.com> <1347373691.24731.125.camel@aleeredhat.laptop> <504F4CE0.9080706@redhat.com> <504F67C5.4000600@redhat.com> <504F86C0.6060907@redhat.com> <5050BB14.3050502@redhat.com> <5053350D.5010805@redhat.com> <50539F44.5040308@redhat.com> <50571879.3080401@redhat.com> Message-ID: <1347896386.2624.8.camel@aleeredhat.laptop> On Mon, 2012-09-17 at 14:32 +0200, Petr Viktorin wrote: > On 09/14/2012 11:19 PM, Rob Crittenden wrote: > > Petr Viktorin wrote: > >> On 09/12/2012 06:40 PM, Petr Viktorin wrote: > >>> A new Dogtag build with changed pkispawn/pkidestroy locations should be > >>> out later today. The attached patch should work with that build. > > > > Fresh install is failing in F-18. > > > > ki-tools-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.i686 > > pki-base-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.noarch > > pki-server-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.noarch > > pki-silent-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.noarch > > pki-symkey-9.0.21-1.fc18.x86_64 > > dogtag-pki-ca-theme-10.0.0-0.1.a1.20120914T0604zgit69c0684.fc18.noarch > > pki-selinux-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.noarch > > pki-ca-10.0.0-0.33.a1.20120914T0536zgit69c0684.fc18.noarch > > pki-setup-9.0.21-1.fc18.noarch > > > > > > rob > > > > > > Ade, your patch adds a step of moving > /var/lib/pki/pki-tomcat/alias/ca_admin_cert.p12 to /root/ca-agent.p12 > right after calling pkispawn. > It seems the file is not created on f18. Did something change in Dogtag > or are we calling it incorrectly? > > Just for the record, as discussed on IRC, the problem is that the latest build of apache-commons-codec (1.6.4) is missing the /usr/share/java/apache-commons-codec.jar symlink. A bug has been filed https://bugzilla.redhat.com/show_bug.cgi?id=857947 and is being addressed. Until a new build of apache-commons-codec is available, you can use the workaround of creating the missing link: ln -s /usr/share/java/commons-codec.jar /usr/share/java/apache-commons-codec.jar Also, there is a bug where pkispawn requires pki-symkey to be installed in order to complete. We will fix pkispawn to not require this - because its only needed for the TKS, and deliver that in a subsequent build. For now though, just make sure pki-symkey is installed. We'll do a new build once you guys have had a chance to try the current build out a bit and report issues. Ade From abokovoy at redhat.com Mon Sep 17 15:44:36 2012 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Mon, 17 Sep 2012 18:44:36 +0300 Subject: [Freeipa-devel] [PATCH] 0073 Add trust verification code Message-ID: <20120917154436.GA12295@redhat.com> Hi, Following patch adds trust verification sequence to the case when we establish trust with knowledge of AD administrative credentials. As we found out, in order to validate/verify trust, one has to have administrative credentials for the trusted domain, since there are few RPCs that should be performed against trusted domain's DC's LSA and NetLogon pipes and these are protected by administrative credentials. Thus, when we know admin credentials for the remote domain, we can perform the trust validation. https://fedorahosted.org/freeipa/ticket/2763 -- / Alexander Bokovoy -------------- next part -------------- >From ddf4205c8b3182cbb19328dc9f8b21ede5de3c65 Mon Sep 17 00:00:00 2001 From: Alexander Bokovoy Date: Thu, 13 Sep 2012 20:01:55 +0300 Subject: [PATCH] Add verification of the AD trust Since we only can perform verification when AD admin credentials are available, report that trust should be verified from the AD side in other cases, including unsuccessful verification. Once trust is added, status of it is never stored anywhere. https://fedorahosted.org/freeipa/ticket/2763 --- ipalib/plugins/trust.py | 12 +++++++----- ipaserver/dcerpc.py | 31 ++++++++++++++++++++++++++++--- 2 files changed, 35 insertions(+), 8 deletions(-) diff --git a/ipalib/plugins/trust.py b/ipalib/plugins/trust.py index 074560dc27eb121b5035ba9a8260e5ab24b2b4b5..2e20725e6343dfd7ea602dd7903745cd0a0e0c62 100644 --- a/ipalib/plugins/trust.py +++ b/ipalib/plugins/trust.py @@ -60,8 +60,8 @@ _trust_type_dict = {1 : _('Non-Active Directory domain'), _trust_direction_dict = {1 : _('Trusting forest'), 2 : _('Trusted forest'), 3 : _('Two-way trust')} -_trust_status = {1 : _('Established and verified'), - 2 : _('Waiting for confirmation by remote side')} +_trust_status_dict = {True : _('Established and verified'), + False : _('Waiting for confirmation by remote side')} _trust_type_dict_unknown = _('Unknown') def trust_type_string(level): @@ -84,7 +84,7 @@ def trust_direction_string(level): return unicode(string) def trust_status_string(level): - string = _trust_direction_dict.get(int(level), _trust_type_dict_unknown) + string = _trust_status_dict.get(level, _trust_type_dict_unknown) return unicode(string) class trust(LDAPObject): @@ -190,6 +190,8 @@ class trust_add(LDAPCreate): result['result'] = trusts[0][1] result['result']['trusttype'] = [trust_type_string(result['result']['ipanttrusttype'][0])] result['result']['trustdirection'] = [trust_direction_string(result['result']['ipanttrustdirection'][0])] + result['result']['truststatus'] = [trust_status_string(result['verified'])] + del result['verified'] return result @@ -272,14 +274,14 @@ class trust_add(LDAPCreate): if result is None: raise errors.ValidationError(name=_('AD Trust setup'), error=_('Unable to verify write permissions to the AD')) - return dict(result=dict(), value=trustinstance.remote_domain.info['dns_domain']) + return dict(value=trustinstance.remote_domain.info['dns_domain'], verified=result['verified']) # 2. We don't have access to the remote domain and trustdom password # is provided. Do the work on our side and inform what to do on remote # side. if 'trust_secret' in options: result = trustinstance.join_ad_ipa_half(keys[-1], realm_server, options['trust_secret']) - return dict(result=dict(), value=trustinstance.remote_domain.info['dns_domain']) + return dict(value=trustinstance.remote_domain.info['dns_domain'], verified=result['verified']) raise errors.ValidationError(name=_('AD Trust setup'), error=_('Not enough arguments specified to perform trust setup')) class trust_del(LDAPDelete): diff --git a/ipaserver/dcerpc.py b/ipaserver/dcerpc.py index b7ccd15d3e9008fddb6dc5419fc05c50ede39d26..86cf01dbac9aca21c35d2db65ef4d4c56e313709 100644 --- a/ipaserver/dcerpc.py +++ b/ipaserver/dcerpc.py @@ -35,7 +35,7 @@ import os, string, struct, copy import uuid from samba import param from samba import credentials -from samba.dcerpc import security, lsa, drsblobs, nbt +from samba.dcerpc import security, lsa, drsblobs, nbt, netlogon from samba.ndr import ndr_pack from samba import net import samba @@ -217,6 +217,7 @@ class TrustDomainInstance(object): if self._pipe is None: raise errors.RemoteRetrieveError( reason=_('Cannot establish LSA connection to %(host)s. Is CIFS server running?') % dict(host=remote_host)) + self.binding = binding def __gen_lsa_bindings(self, remote_host): """ @@ -251,6 +252,7 @@ class TrustDomainInstance(object): self.info['dns_domain'] = unicode(result.dns_domain) self.info['dns_forest'] = unicode(result.forest) self.info['guid'] = unicode(result.domain_uuid) + self.info['dc'] = unicode(result.pdc_dns_name) # Netlogon response doesn't contain SID of the domain. # We need to do rootDSE search with LDAP_SERVER_EXTENDED_DN_OID control to reveal the SID @@ -291,6 +293,7 @@ class TrustDomainInstance(object): self.info['dns_forest'] = unicode(result.dns_forest.string) self.info['guid'] = unicode(result.domain_guid) self.info['sid'] = unicode(result.sid) + self.info['dc'] = remote_host def generate_auth(self, trustdom_secret): def arcfour_encrypt(key, data): @@ -374,6 +377,27 @@ class TrustDomainInstance(object): except RuntimeError, (num, message): raise assess_dcerpc_exception(num=num, message=message) + def verify_trust(self, another_domain): + def retrieve_netlogon_info_2(domain, function_code, data): + try: + netr_pipe = netlogon.netlogon(domain.binding, domain.parm, domain.creds) + result = netr_pipe.netr_LogonControl2Ex(logon_server=None, + function_code=function_code, + level=2, + data=data + ) + return result + except RuntimeError, (num, message): + raise assess_dcerpc_exception(num=num, message=message) + + result = retrieve_netlogon_info_2(self, + netlogon.NETLOGON_CONTROL_TC_VERIFY, + another_domain.info['dns_domain']) + if (unicode(result.trusted_dc_name)[2:] == another_domain.info['dc'] and + result.tc_connection_status == (0, 'WERR_OK')): + return True + return False + class TrustDomainJoins(object): def __init__(self, api): self.api = api @@ -447,7 +471,8 @@ class TrustDomainJoins(object): trustdom_pass = samba.generate_random_password(128, 128) self.remote_domain.establish_trust(self.local_domain, trustdom_pass) self.local_domain.establish_trust(self.remote_domain, trustdom_pass) - return dict(local=self.local_domain, remote=self.remote_domain) + result = self.remote_domain.verify_trust(self.local_domain) + return dict(local=self.local_domain, remote=self.remote_domain, verified=result) return None def join_ad_ipa_half(self, realm, realm_server, trustdom_passwd): @@ -456,4 +481,4 @@ class TrustDomainJoins(object): self.__populate_remote_domain(realm, realm_server, realm_passwd=None) self.local_domain.establish_trust(self.remote_domain, trustdom_passwd) - return dict(local=self.local_domain, remote=self.remote_domain) + return dict(local=self.local_domain, remote=self.remote_domain, verified=False) -- 1.7.12 From mkosek at redhat.com Mon Sep 17 15:49:18 2012 From: mkosek at redhat.com (Martin Kosek) Date: Mon, 17 Sep 2012 17:49:18 +0200 Subject: [Freeipa-devel] [PATCH] 1031 run cleanallruv task In-Reply-To: <50573082.3020601@redhat.com> References: <4FF3047C.9020202@redhat.com> <4FF41C8F.2020003@redhat.com> <4FF5DF79.6090806@redhat.com> <4FFBCC38.7080105@redhat.com> <5037B79E.3070603@redhat.com> <50479491.1060200@redhat.com> <50488DD0.1080507@redhat.com> <5048BEF8.5000401@redhat.com> <5048C46A.8020207@redhat.com> <5048C754.2030708@redhat.com> <5048C9BE.1050005@redhat.com> <504912F9.1020907@redhat.com> <50531DDC.5050007@redhat.com> <505382CF.1030901@redhat.com> <5056E754.1080201@redhat.com> <50572DEF.7020104@redhat.com> <50572F9A.3040605@redhat.com> <50573082.3020601@redhat.com> Message-ID: <5057467E.6040209@redhat.com> On 09/17/2012 04:15 PM, Rob Crittenden wrote: > Martin Kosek wrote: >> On 09/17/2012 04:04 PM, Rob Crittenden wrote: >>> Martin Kosek wrote: >>>> On 09/14/2012 09:17 PM, Rob Crittenden wrote: >>>>> Martin Kosek wrote: >>>>>> On 09/06/2012 11:17 PM, Rob Crittenden wrote: >>>>>>> Martin Kosek wrote: >>>>>>>> On 09/06/2012 05:55 PM, Rob Crittenden wrote: >>>>>>>>> Rob Crittenden wrote: >>>>>>>>>> Rob Crittenden wrote: >>>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>>> On 09/05/2012 08:06 PM, Rob Crittenden wrote: >>>>>>>>>>>>> Rob Crittenden wrote: >>>>>>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>>>>>> On 07/05/2012 08:39 PM, Rob Crittenden wrote: >>>>>>>>>>>>>>>> Martin Kosek wrote: >>>>>>>>>>>>>>>>> On 07/03/2012 04:41 PM, Rob Crittenden wrote: >>>>>>>>>>>>>>>>>> Deleting a replica can leave a replication vector (RUV) on the >>>>>>>>>>>>>>>>>> other servers. >>>>>>>>>>>>>>>>>> This can confuse things if the replica is re-added, and it also >>>>>>>>>>>>>>>>>> causes the >>>>>>>>>>>>>>>>>> server to calculate changes against a server that may no longer >>>>>>>>>>>>>>>>>> exist. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 389-ds-base provides a new task that self-propogates itself >>>>>>>>>>>>>>>>>> to all >>>>>>>>>>>>>>>>>> available >>>>>>>>>>>>>>>>>> replicas to clean this RUV data. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> This patch will create this task at deletion time to hopefully >>>>>>>>>>>>>>>>>> clean things up. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> It isn't perfect. If any replica is down or unavailable at the >>>>>>>>>>>>>>>>>> time >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> cleanruv task fires, and then comes back up, the old RUV data >>>>>>>>>>>>>>>>>> may be >>>>>>>>>>>>>>>>>> re-propogated around. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> To make things easier in this case I've added two new >>>>>>>>>>>>>>>>>> commands to >>>>>>>>>>>>>>>>>> ipa-replica-manage. The first lists the replication ids of >>>>>>>>>>>>>>>>>> all the >>>>>>>>>>>>>>>>>> servers we >>>>>>>>>>>>>>>>>> have a RUV for. Using this you can call clean_ruv with the >>>>>>>>>>>>>>>>>> replication id of a >>>>>>>>>>>>>>>>>> server that no longer exists to try the cleanallruv step again. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> This is quite dangerous though. If you run cleanruv against a >>>>>>>>>>>>>>>>>> replica id that >>>>>>>>>>>>>>>>>> does exist it can cause a loss of data. I believe I've put in >>>>>>>>>>>>>>>>>> enough scary >>>>>>>>>>>>>>>>>> warnings about this. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> rob >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Good work there, this should make cleaning RUVs much easier than >>>>>>>>>>>>>>>>> with the >>>>>>>>>>>>>>>>> previous version. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> This is what I found during review: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1) list_ruv and clean_ruv command help in man is quite lost. I >>>>>>>>>>>>>>>>> think >>>>>>>>>>>>>>>>> it would >>>>>>>>>>>>>>>>> help if we for example have all info for commands indented. This >>>>>>>>>>>>>>>>> way >>>>>>>>>>>>>>>>> user could >>>>>>>>>>>>>>>>> simply over-look the new commands in the man page. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2) I would rename new commands to clean-ruv and list-ruv to make >>>>>>>>>>>>>>>>> them >>>>>>>>>>>>>>>>> consistent with the rest of the commands (re-initialize, >>>>>>>>>>>>>>>>> force-sync). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 3) It would be nice to be able to run clean_ruv command in an >>>>>>>>>>>>>>>>> unattended way >>>>>>>>>>>>>>>>> (for better testing), i.e. respect --force option as we already >>>>>>>>>>>>>>>>> do for >>>>>>>>>>>>>>>>> ipa-replica-manage del. This fix would aid test automation in the >>>>>>>>>>>>>>>>> future. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 4) (minor) The new question (and the del too) does not react too >>>>>>>>>>>>>>>>> well for >>>>>>>>>>>>>>>>> CTRL+D: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> # ipa-replica-manage clean_ruv 3 --force >>>>>>>>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>>>>>>>> Continue to clean? [no]: unexpected error: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 5) Help for clean_ruv command without a required parameter is >>>>>>>>>>>>>>>>> quite >>>>>>>>>>>>>>>>> confusing >>>>>>>>>>>>>>>>> as it reports that command is wrong and not the parameter: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> # ipa-replica-manage clean_ruv >>>>>>>>>>>>>>>>> Usage: ipa-replica-manage [options] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ipa-replica-manage: error: must provide a command [clean_ruv | >>>>>>>>>>>>>>>>> force-sync | >>>>>>>>>>>>>>>>> disconnect | connect | del | re-initialize | list | list_ruv] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> It seems you just forgot to specify the error message in the >>>>>>>>>>>>>>>>> command >>>>>>>>>>>>>>>>> definition >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 6) When the remote replica is down, the clean_ruv command fails >>>>>>>>>>>>>>>>> with an >>>>>>>>>>>>>>>>> unexpected error: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [root at vm-086 ~]# ipa-replica-manage clean_ruv 5 >>>>>>>>>>>>>>>>> Clean the Replication Update Vector for >>>>>>>>>>>>>>>>> vm-055.idm.lab.bos.redhat.com:389 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>>>>>>>>>>> longer replicate so it may miss updates while the process >>>>>>>>>>>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>>>>>>>>>>> consistency. Be very careful. >>>>>>>>>>>>>>>>> Continue to clean? [no]: y >>>>>>>>>>>>>>>>> unexpected error: {'desc': 'Operations error'} >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> /var/log/dirsrv/slapd-IDM-LAB-BOS-REDHAT-COM/errors: >>>>>>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>>>>>> cleanAllRUV_task: failed >>>>>>>>>>>>>>>>> to connect to repl agreement connection >>>>>>>>>>>>>>>>> (cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> tree,cn=config), error 105 >>>>>>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>>>>>> cleanAllRUV_task: replica >>>>>>>>>>>>>>>>> (cn=meTovm-055.idm.lab. >>>>>>>>>>>>>>>>> bos.redhat.com,cn=replica,cn=dc\3Didm\2Cdc\3Dlab\2Cdc\3Dbos\2Cdc\3Dredhat\2Cdc\3Dcom,cn=mapping >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> tree, cn=config) has not been cleaned. You will need to rerun >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> CLEANALLRUV task on this replica. >>>>>>>>>>>>>>>>> [04/Jul/2012:06:28:16 -0400] NSMMReplicationPlugin - >>>>>>>>>>>>>>>>> cleanAllRUV_task: Task >>>>>>>>>>>>>>>>> failed (1) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> In this case I think we should inform user that the command >>>>>>>>>>>>>>>>> failed, >>>>>>>>>>>>>>>>> possibly >>>>>>>>>>>>>>>>> because of disconnected replicas and that they could enable the >>>>>>>>>>>>>>>>> replicas and >>>>>>>>>>>>>>>>> try again. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 7) (minor) "pass" is now redundant in replication.py: >>>>>>>>>>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>>>>>>>>>>> + # We can't make the server we're removing read-only >>>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>> + # this isn't a show-stopper >>>>>>>>>>>>>>>>> + root_logger.debug("No permission to switch >>>>>>>>>>>>>>>>> replica to >>>>>>>>>>>>>>>>> read-only, >>>>>>>>>>>>>>>>> continuing anyway") >>>>>>>>>>>>>>>>> + pass >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I think this addresses everything. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> rob >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, almost there! I just found one more issue which needs to be >>>>>>>>>>>>>>> fixed >>>>>>>>>>>>>>> before we push: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>>>>>>>>>>>>> Directory Manager password: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing >>>>>>>>>>>>>>> removal >>>>>>>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>>>>>>>>> "Can't >>>>>>>>>>>>>>> contact LDAP server"} >>>>>>>>>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> There were issues removing a connection: %d format: a number is >>>>>>>>>>>>>>> required, not str >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Failed to get data from 'vm-055.idm.lab.bos.redhat.com': {'desc': >>>>>>>>>>>>>>> "Can't >>>>>>>>>>>>>>> contact LDAP server"} >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a traceback I retrieved: >>>>>>>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>>>>>> File "/sbin/ipa-replica-manage", line 425, in del_master >>>>>>>>>>>>>>> del_link(realm, r, hostname, options.dirman_passwd, >>>>>>>>>>>>>>> force=True) >>>>>>>>>>>>>>> File "/sbin/ipa-replica-manage", line 271, in del_link >>>>>>>>>>>>>>> repl1.cleanallruv(replica_id) >>>>>>>>>>>>>>> File >>>>>>>>>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> line 1094, in cleanallruv >>>>>>>>>>>>>>> root_logger.debug("Creating CLEANALLRUV task for >>>>>>>>>>>>>>> replica id >>>>>>>>>>>>>>> %d" % >>>>>>>>>>>>>>> replicaId) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The problem here is that you don't convert replica_id to int in >>>>>>>>>>>>>>> this >>>>>>>>>>>>>>> part: >>>>>>>>>>>>>>> + replica_id = None >>>>>>>>>>>>>>> + if repl2: >>>>>>>>>>>>>>> + replica_id = repl2._get_replica_id(repl2.conn, None) >>>>>>>>>>>>>>> + else: >>>>>>>>>>>>>>> + servers = get_ruv(realm, replica1, dirman_passwd) >>>>>>>>>>>>>>> + for (netloc, rid) in servers: >>>>>>>>>>>>>>> + if netloc.startswith(replica2): >>>>>>>>>>>>>>> + replica_id = rid >>>>>>>>>>>>>>> + break >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Martin >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Updated patch using new mechanism in 389-ds-base. This should more >>>>>>>>>>>>>> thoroughly clean out RUV data when a replica is being deleted, and >>>>>>>>>>>>>> provide for a way to delete RUV data afterwards too if necessary. >>>>>>>>>>>>>> >>>>>>>>>>>>>> rob >>>>>>>>>>>>> >>>>>>>>>>>>> Rebased patch >>>>>>>>>>>>> >>>>>>>>>>>>> rob >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 0) As I wrote in a review for your patch 1041, changelog entry slipped >>>>>>>>>>>> elsewhere. >>>>>>>>>>>> >>>>>>>>>>>> 1) The following KeyboardInterrupt except class looks suspicious. I >>>>>>>>>>>> know why >>>>>>>>>>>> you have it there, but since it is generally a bad thing to do, some >>>>>>>>>>>> comment >>>>>>>>>>>> why it is needed would be useful. >>>>>>>>>>>> >>>>>>>>>>>> @@ -256,6 +263,17 @@ def del_link(realm, replica1, replica2, >>>>>>>>>>>> dirman_passwd, >>>>>>>>>>>> force=False): >>>>>>>>>>>> repl1.delete_agreement(replica2) >>>>>>>>>>>> repl1.delete_referral(replica2) >>>>>>>>>>>> >>>>>>>>>>>> + if type1 == replication.IPA_REPLICA: >>>>>>>>>>>> + if repl2: >>>>>>>>>>>> + ruv = repl2._get_replica_id(repl2.conn, None) >>>>>>>>>>>> + else: >>>>>>>>>>>> + ruv = get_ruv_by_host(realm, replica1, replica2, >>>>>>>>>>>> dirman_passwd) >>>>>>>>>>>> + >>>>>>>>>>>> + try: >>>>>>>>>>>> + repl1.cleanallruv(ruv) >>>>>>>>>>>> + except KeyboardInterrupt: >>>>>>>>>>>> + pass >>>>>>>>>>>> + >>>>>>>>>>>> >>>>>>>>>>>> Maybe you just wanted to do some cleanup and then "raise" again? >>>>>>>>>>> >>>>>>>>>>> No, it is there because it is safe to break out of it. The task will >>>>>>>>>>> continue to run. I added some verbiage. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2) This is related to 1), but when some replica is down, >>>>>>>>>>>> "ipa-replica-manage >>>>>>>>>>>> del" may wait indefinitely when some remote replica is down, right? >>>>>>>>>>>> >>>>>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>>>>>>>> Deleting a master is irreversible. >>>>>>>>>>>> To reconnect to the remote master you will need to prepare a new >>>>>>>>>>>> replica file >>>>>>>>>>>> and re-install. >>>>>>>>>>>> Continue to delete? [no]: y >>>>>>>>>>>> ipa: INFO: Setting agreement >>>>>>>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>>>>>>>>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>>>>>>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> tree,cn=config >>>>>>>>>>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica >>>>>>>>>>>> acquired >>>>>>>>>>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>>>>>>>>>> Background task created to clean replication data >>>>>>>>>>>> >>>>>>>>>>>> ... after about a minute I hit CTRL+C >>>>>>>>>>>> >>>>>>>>>>>> ^CDeleted replication agreement from >>>>>>>>>>>> 'vm-086.idm.lab.bos.redhat.com' to >>>>>>>>>>>> 'vm-055.idm.lab.bos.redhat.com' >>>>>>>>>>>> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: NS record >>>>>>>>>>>> does not >>>>>>>>>>>> contain 'vm-055.idm.lab.bos.redhat.com.' >>>>>>>>>>>> You may need to manually remove them from the tree >>>>>>>>>>>> >>>>>>>>>>>> I think it would be better to inform user that some remote replica is >>>>>>>>>>>> down or >>>>>>>>>>>> at least that we are waiting for the task to complete. Something like >>>>>>>>>>>> that: >>>>>>>>>>>> >>>>>>>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>>>>>>>>>> ... >>>>>>>>>>>> Background task created to clean replication data >>>>>>>>>>>> Replication data clean up may take very long time if some replica is >>>>>>>>>>>> unreachable >>>>>>>>>>>> Hit CTRL+C to interrupt the wait >>>>>>>>>>>> ^C Clean up wait interrupted >>>>>>>>>>>> .... >>>>>>>>>>>> [continue with del] >>>>>>>>>>> >>>>>>>>>>> Yup, did this in #1. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 3) (minor) When there is a cleanruv task running and you run >>>>>>>>>>>> "ipa-replica-manage del", there is a unexpected error message with >>>>>>>>>>>> duplicate >>>>>>>>>>>> task object in LDAP: >>>>>>>>>>>> >>>>>>>>>>>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com --force >>>>>>>>>>>> Unable to connect to replica vm-072.idm.lab.bos.redhat.com, forcing >>>>>>>>>>>> removal >>>>>>>>>>>> FAIL >>>>>>>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': >>>>>>>>>>>> "Can't >>>>>>>>>>>> contact LDAP server"} >>>>>>>>>>>> Forcing removal on 'vm-086.idm.lab.bos.redhat.com' >>>>>>>>>>>> >>>>>>>>>>>> There were issues removing a connection: This entry already exists >>>>>>>>>>>> <<<<<<<<< >>>>>>>>>>>> >>>>>>>>>>>> Failed to get data from 'vm-072.idm.lab.bos.redhat.com': {'desc': >>>>>>>>>>>> "Can't >>>>>>>>>>>> contact LDAP server"} >>>>>>>>>>>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record >>>>>>>>>>>> does not >>>>>>>>>>>> contain 'vm-072.idm.lab.bos.redhat.com.' >>>>>>>>>>>> You may need to manually remove them from the tree >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I think it should be enough to just catch for "entry already >>>>>>>>>>>> exists" in >>>>>>>>>>>> cleanallruv function, and in such case print a relevant error message >>>>>>>>>>>> bail out. >>>>>>>>>>>> Thus, self.conn.checkTask(dn, dowait=True) would not be called too. >>>>>>>>>>> >>>>>>>>>>> Good catch, fixed. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 4) (minor): In make_readonly function, there is a redundant "pass" >>>>>>>>>>>> statement: >>>>>>>>>>>> >>>>>>>>>>>> + def make_readonly(self): >>>>>>>>>>>> + """ >>>>>>>>>>>> + Make the current replication agreement read-only. >>>>>>>>>>>> + """ >>>>>>>>>>>> + dn = DN(('cn', 'userRoot'), ('cn', 'ldbm database'), >>>>>>>>>>>> + ('cn', 'plugins'), ('cn', 'config')) >>>>>>>>>>>> + >>>>>>>>>>>> + mod = [(ldap.MOD_REPLACE, 'nsslapd-readonly', 'on')] >>>>>>>>>>>> + try: >>>>>>>>>>>> + self.conn.modify_s(dn, mod) >>>>>>>>>>>> + except ldap.INSUFFICIENT_ACCESS: >>>>>>>>>>>> + # We can't make the server we're removing read-only but >>>>>>>>>>>> + # this isn't a show-stopper >>>>>>>>>>>> + root_logger.debug("No permission to switch replica to >>>>>>>>>>>> read-only, >>>>>>>>>>>> continuing anyway") >>>>>>>>>>>> + pass <<<<<<<<<<<<<<< >>>>>>>>>>> >>>>>>>>>>> Yeah, this is one of my common mistakes. I put in a pass initially, >>>>>>>>>>> then >>>>>>>>>>> add logging in front of it and forget to delete the pass. Its gone now. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 5) In clean_ruv, I think allowing a --force option to bypass the >>>>>>>>>>>> user_input >>>>>>>>>>>> would be helpful (at least for test automation): >>>>>>>>>>>> >>>>>>>>>>>> + if not ipautil.user_input("Continue to clean?", False): >>>>>>>>>>>> + sys.exit("Aborted") >>>>>>>>>>> >>>>>>>>>>> Yup, added. >>>>>>>>>>> >>>>>>>>>>> rob >>>>>>>>>> >>>>>>>>>> Slightly revised patch. I still had a window open with one unsaved >>>>>>>>>> change. >>>>>>>>>> >>>>>>>>>> rob >>>>>>>>>> >>>>>>>>> >>>>>>>>> Apparently there were two unsaved changes, one of which was lost. This >>>>>>>>> adds in >>>>>>>>> the 'entry already exists' fix. >>>>>>>>> >>>>>>>>> rob >>>>>>>>> >>>>>>>> >>>>>>>> Just one last thing (otherwise the patch is OK) - I don't think this is >>>>>>>> what we >>>>>>>> want :-) >>>>>>>> >>>>>>>> # ipa-replica-manage clean-ruv 8 >>>>>>>> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 >>>>>>>> >>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>> longer replicate so it may miss updates while the process >>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>> consistency. Be very careful. >>>>>>>> Continue to clean? [no]: y <<<<<< >>>>>>>> Aborted >>>>>>>> >>>>>>>> >>>>>>>> Nor this exception, (your are checking for wrong exception): >>>>>>>> >>>>>>>> # ipa-replica-manage clean-ruv 8 >>>>>>>> Clean the Replication Update Vector for vm-055.idm.lab.bos.redhat.com:389 >>>>>>>> >>>>>>>> Cleaning the wrong replica ID will cause that server to no >>>>>>>> longer replicate so it may miss updates while the process >>>>>>>> is running. It would need to be re-initialized to maintain >>>>>>>> consistency. Be very careful. >>>>>>>> Continue to clean? [no]: >>>>>>>> unexpected error: This entry already exists >>>>>>>> >>>>>>>> This is the exception: >>>>>>>> >>>>>>>> Traceback (most recent call last): >>>>>>>> File "/sbin/ipa-replica-manage", line 651, in >>>>>>>> main() >>>>>>>> File "/sbin/ipa-replica-manage", line 648, in main >>>>>>>> clean_ruv(realm, args[1], options) >>>>>>>> File "/sbin/ipa-replica-manage", line 373, in clean_ruv >>>>>>>> thisrepl.cleanallruv(ruv) >>>>>>>> File >>>>>>>> "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", >>>>>>>> line 1136, in cleanallruv >>>>>>>> self.conn.addEntry(e) >>>>>>>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line >>>>>>>> 503, in >>>>>>>> addEntry >>>>>>>> self.__handle_errors(e, arg_desc=arg_desc) >>>>>>>> File "/usr/lib/python2.7/site-packages/ipaserver/ipaldap.py", line >>>>>>>> 321, in >>>>>>>> __handle_errors >>>>>>>> raise errors.DuplicateEntry() >>>>>>>> ipalib.errors.DuplicateEntry: This entry already exists >>>>>>>> >>>>>>>> Martin >>>>>>>> >>>>>>> >>>>>>> Fixed that and a couple of other problems. When doing a disconnect we >>>>>>> should >>>>>>> not also call clean-ruv. >>>>>> >>>>>> Ah, good self-catch. >>>>>> >>>>>>> >>>>>>> I also got tired of seeing crappy error messages so I added a little >>>>>>> convert >>>>>>> utility. >>>>>>> >>>>>>> rob >>>>>> >>>>>> 1) There is CLEANALLRUV stuff included in 1050-3 and not here. There are >>>>>> also >>>>>> some finding for this new code. >>>>>> >>>>>> >>>>>> 2) We may want to bump Requires to higher version of 389-ds-base >>>>>> (389-ds-base-1.2.11.14-1) - it contains a fix for CLEANALLRUV+winsync bug I >>>>>> found earlier. >>>>>> >>>>>> >>>>>> 3) I just discovered another suspicious behavior. When we are deleting a >>>>>> master >>>>>> that has links also to other master(s) we delete those too. But we also >>>>>> automatically run CLEANALLRUV in these cases, so we may end up in multiple >>>>>> tasks being started on different masters - this does not look right. >>>>>> >>>>>> I think we may rather want to at first delete all links and then run >>>>>> CLEANALLRUV task, just for one time. This is what I get with current code: >>>>>> >>>>>> # ipa-replica-manage del vm-072.idm.lab.bos.redhat.com >>>>>> Directory Manager password: >>>>>> >>>>>> Deleting a master is irreversible. >>>>>> To reconnect to the remote master you will need to prepare a new replica >>>>>> file >>>>>> and re-install. >>>>>> Continue to delete? [no]: yes >>>>>> ipa: INFO: Setting agreement >>>>>> cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>> >>>>>> >>>>>> >>>>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>>>> cn=meTovm-055.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>> >>>>>> >>>>>> >>>>>> tree,cn=config >>>>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica acquired >>>>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>>>> Background task created to clean replication data. This may take a while. >>>>>> This may be safely interrupted with Ctrl+C >>>>>> >>>>>> ^CWait for task interrupted. It will continue to run in the background >>>>>> >>>>>> Deleted replication agreement from 'vm-055.idm.lab.bos.redhat.com' to >>>>>> 'vm-072.idm.lab.bos.redhat.com' >>>>>> ipa: INFO: Setting agreement >>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>> >>>>>> >>>>>> >>>>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>>>> cn=meTovm-086.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>>>> >>>>>> >>>>>> >>>>>> tree,cn=config >>>>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica acquired >>>>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>>>> Background task created to clean replication data. This may take a while. >>>>>> This may be safely interrupted with Ctrl+C >>>>>> >>>>>> ^CWait for task interrupted. It will continue to run in the background >>>>>> >>>>>> Deleted replication agreement from 'vm-086.idm.lab.bos.redhat.com' to >>>>>> 'vm-072.idm.lab.bos.redhat.com' >>>>>> Failed to cleanup vm-072.idm.lab.bos.redhat.com DNS entries: NS record does >>>>>> not >>>>>> contain 'vm-072.idm.lab.bos.redhat.com.' >>>>>> You may need to manually remove them from the tree >>>>>> >>>>>> Martin >>>>>> >>>>> >>>>> All issues addressed and I pulled in abort-clean-ruv from 1050. I added a >>>>> list-clean-ruv command as well. >>>>> >>>>> rob >>>> >>>> 1) Patch 1031-9 needs to get squashed with 1031-8 >>>> >>>> >>>> 2) Patch needs a rebase (conflict in freeipa.spec.in) >>>> >>>> >>>> 3) New list-clean-ruv man entry is not right: >>>> >>>> list-clean-ruv [REPLICATION_ID] >>>> - List all running CLEANALLRUV and abort CLEANALLRUV tasks. >>>> >>>> REPLICATION_ID is not its argument. >>> >>> Fixed 1-3. >>> >>>> Btw. new list-clean-ruv command proved very useful for me. >>>> >>>> 4) I just found out we need to do a better job with make_readonly() command. I >>>> get into trouble when disconnecting one link to a remote replica as it was >>>> marked readonly and then I was then unable to manage the disconnected replica >>>> properly (vm-072 is the replica made readonly): >>> >>> Ok, I reset read-only after we delete the agreements. That fixed things up for >>> me. I disconnected a replica and was able to modify entries on that replica >>> afterwards. >>> >>> This affected the --cleanup command too, it would otherwise have succeeded I >>> think. >>> >>> I tested with an A - B - C - A agreement loop. I disconnected A and C and >>> confirmed I could still update entries on C. Then I deleted C, then B, and made >>> sure output looked right, I could still manage entries, etc. >>> >>> rob >>> >>>> >>>> [root at vm-055 ~]# ipa-replica-manage disconnect vm-072.idm.lab.bos.redhat.com >>>> >>>> [root at vm-072 ~]# ipa-replica-manage del vm-055.idm.lab.bos.redhat.com >>>> Deleting a master is irreversible. >>>> To reconnect to the remote master you will need to prepare a new replica file >>>> and re-install. >>>> Continue to delete? [no]: yes >>>> Deleting replication agreements between vm-055.idm.lab.bos.redhat.com and >>>> vm-072.idm.lab.bos.redhat.com >>>> ipa: INFO: Setting agreement >>>> cn=meTovm-072.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>> >>>> >>>> tree,cn=config schedule to 2358-2359 0 to force synch >>>> ipa: INFO: Deleting schedule 2358-2359 0 from agreement >>>> cn=meTovm-072.idm.lab.bos.redhat.com,cn=replica,cn=dc\=idm\,dc\=lab\,dc\=bos\,dc\=redhat\,dc\=com,cn=mapping >>>> >>>> >>>> tree,cn=config >>>> ipa: INFO: Replication Update in progress: FALSE: status: 0 Replica acquired >>>> successfully: Incremental update succeeded: start: 0: end: 0 >>>> Deleted replication agreement from 'vm-072.idm.lab.bos.redhat.com' to >>>> 'vm-055.idm.lab.bos.redhat.com' >>>> Unable to remove replication agreement for vm-055.idm.lab.bos.redhat.com from >>>> vm-072.idm.lab.bos.redhat.com. >>>> Background task created to clean replication data. This may take a while. >>>> This may be safely interrupted with Ctrl+C >>>> ^CWait for task interrupted. It will continue to run in the background >>>> >>>> Failed to cleanup vm-055.idm.lab.bos.redhat.com entries: Server is >>>> unwilling to >>>> perform: database is read-only arguments: >>>> dn=krbprincipalname=ldap/vm-055.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM,cn=services,cn=accounts,dc=idm,dc=lab,dc=bos,dc=redhat,dc=com >>>> >>>> >>>> >>>> You may need to manually remove them from the tree >>>> ipa: INFO: Unhandled LDAPError: {'info': 'database is read-only', 'desc': >>>> 'Server is unwilling to perform'} >>>> >>>> Failed to cleanup vm-055.idm.lab.bos.redhat.com DNS entries: Server is >>>> unwilling to perform: database is read-only >>>> >>>> You may need to manually remove them from the tree >>>> >>>> >>>> --cleanup did not work for me as well: >>>> [root at vm-072 ~]# ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>> --cleanup >>>> Cleaning a master is irreversible. >>>> This should not normally be require, so use cautiously. >>>> Continue to clean master? [no]: yes >>>> unexpected error: Server is unwilling to perform: database is read-only >>>> arguments: >>>> dn=krbprincipalname=ldap/vm-055.idm.lab.bos.redhat.com at IDM.LAB.BOS.REDHAT.COM,cn=services,cn=accounts,dc=idm,dc=lab,dc=bos,dc=redhat,dc=com >>>> >>>> >>>> >>>> Martin >>>> >>> >> >> I think you sent a wrong patch... >> >> Martin >> > > I hate Mondays. > > rob Maybe you will like this one a little bit more :-) ACK. Pushed to master, ipa-3-0. Martin From mkosek at redhat.com Mon Sep 17 15:59:20 2012 From: mkosek at redhat.com (Martin Kosek) Date: Mon, 17 Sep 2012 17:59:20 +0200 Subject: [Freeipa-devel] [PATCH] 1050 prevent replica orphans In-Reply-To: <50572E6D.6070201@redhat.com> References: <5040F6E2.7060109@redhat.com> <5040F722.4060208@redhat.com> <5048BFAD.1070807@redhat.com> <50491414.2070504@redhat.com> <1347008704.2829.13.camel@priserak> <504E329D.6070905@redhat.com> <50531B7C.8070401@redhat.com> <50538284.8090205@redhat.com> <50572E6D.6070201@redhat.com> Message-ID: <505748D8.1070407@redhat.com> On 09/17/2012 04:06 PM, Martin Kosek wrote: > On 09/14/2012 09:16 PM, Rob Crittenden wrote: >> Martin Kosek wrote: >>> On 09/10/2012 08:34 PM, Rob Crittenden wrote: >>>> Martin Kosek wrote: >>>>> On Thu, 2012-09-06 at 17:22 -0400, Rob Crittenden wrote: >>>>>> Martin Kosek wrote: >>>>>>> On 08/31/2012 07:40 PM, Rob Crittenden wrote: >>>>>>>> Rob Crittenden wrote: >>>>>>>>> It was possible use ipa-replica-manage connect/disconnect/del to end up >>>>>>>>> orphaning or or more IPA masters. This is an attempt to catch and >>>>>>>>> prevent that case. >>>>>>>>> >>>>>>>>> I tested with this topology, trying to delete B. >>>>>>>>> >>>>>>>>> A <-> B <-> C >>>>>>>>> >>>>>>>>> I got here by creating B and C from A, connecting B to C then deleting >>>>>>>>> the link from A to B, so it went from A -> B and A -> C to the above. >>>>>>>>> >>>>>>>>> What I do is look up the servers that the delete candidate host has >>>>>>>>> connections to and see if we're the last link. >>>>>>>>> >>>>>>>>> I added an escape clause if there are only two masters. >>>>>>>>> >>>>>>>>> rob >>>>>>>> >>>>>>>> Oh, this relies on my cleanruv patch 1031. >>>>>>>> >>>>>>>> rob >>>>>>>> >>>>>>> >>>>>>> 1) When I run ipa-replica-manage del --force to an already uninstalled host, >>>>>>> the new code will prevent me the deletation because it cannot connect to >>>>>>> it. It >>>>>>> also crashes with UnboundLocalError: >>>>>>> >>>>>>> # ipa-replica-manage del vm-055.idm.lab.bos.redhat.com --force >>>>>>> >>>>>>> Unable to connect to replica vm-055.idm.lab.bos.redhat.com, forcing removal >>>>>>> Traceback (most recent call last): >>>>>>> File "/sbin/ipa-replica-manage", line 708, in >>>>>>> main() >>>>>>> File "/sbin/ipa-replica-manage", line 677, in main >>>>>>> del_master(realm, args[1], options) >>>>>>> File "/sbin/ipa-replica-manage", line 476, in del_master >>>>>>> sys.exit("Failed read master data from '%s': %s" % (delrepl.hostname, >>>>>>> str(e))) >>>>>>> UnboundLocalError: local variable 'delrepl' referenced before assignment >>>>>> >>>>>> Fixed. >>>>>> >>>>>>> >>>>>>> >>>>>>> I also hit this error when removing a winsync replica. >>>>>> >>>>>> Fixed. >>>>>> >>>>>>> >>>>>>> >>>>>>> 2) As I wrote before, I think having --force option override the user >>>>>>> inquiries >>>>>>> would benefit test automation: >>>>>>> >>>>>>> + if not ipautil.user_input("Continue to delete?", False): >>>>>>> + sys.exit("Aborted") >>>>>> >>>>>> Fixed. >>>>>> >>>>>>> >>>>>>> >>>>>>> 3) I don't think this code won't cover this topology: >>>>>>> >>>>>>> A - B - C - D - E >>>>>>> >>>>>>> It would allow you deleting a replica C even though it would separate A-B >>>>>>> and >>>>>>> D-E. Though we may not want to cover this situation now, what you got is >>>>>>> definitely helping. >>>>>> >>>>>> I think you may be right. I only tested with 4 servers. With this B and >>>>>> D would both still have 2 agreements so wouldn't be covered by the last >>>>>> link test. >>>>> >>>>> Everything looks good now, so ACK. We just need to push it along with >>>>> CLEANALLRUV patch. >>>>> >>>>> Martin >>>>> >>>> >>>> Not to look a gift ACK In the mouth but here is a revised patch. I've added a >>>> cleanup routine to remove an orphaned master properly. If you had tried the >>>> mechanism I outlined in the man page it would have worked but was >>>> less-than-complete. This way is better, just don't try it on a live master. >>>> >>>> I also added a cleanruv abort command, in case you want to kill an existing >>>> task. >>>> >>>> rob >>>> >>> >>> 1) CLEANRUV stuff should be in your patch 1031 and not here (but I will comment >>> on the code in this mail since it is in this patch) >>> >>> >>> 2) In new command defitinion: >>> >>> + "abort-clean-ruv":(1, 1, "Replica ID of to abort cleaning", ""), >>> >>> I miss error message in case REPLICA_ID is not passed, this way error message >>> when I omit the parameter is confusing: >>> >>> # ipa-replica-manage abort-clean-ruv >>> Usage: ipa-replica-manage [options] >>> >>> ipa-replica-manage: error: must provide a command [force-sync | clean-ruv | >>> disconnect | connect | list-ruv | del | re-initialize | list | abort-clean-ruv] >>> >>> On another note, I am thinking about the new command(s). Since we now have >>> abort-clean-ruv command, we may want to also implement list-clean-ruv commands >>> in the future to see all CLEANALLRUV commands in process... Or we may enhance >>> list-ruv to write a flag like [CLEANALLRUV in process] for RUV's for which >>> CLEANALLRUV task is in process. >>> >>> >>> 3) Will clean-ruv - abort-clean-ruv - clean-ruv sequence work? If the aborted >>> CLEANALLRUV task stays in DIT, we may not be able to enter new CLEANALLRUV task >>> because we always use the same DN... >>> >>> Btw. did abort CLEANALLRUV command worked for you? Mine seemed to be stuck on >>> replicas that are down just like CLEANALLRUV command. I had then paralelly >>> running CLEANALLRUV and ABORT CLEANALLRUV running for the same RUV ID. Then, it >>> is unclear to me what this command is actually good for. >>> >>> >>> 4) Man page now contains non-existent command: >>> >>> --- a/install/tools/man/ipa-replica-manage.1 >>> +++ b/install/tools/man/ipa-replica-manage.1 >>> @@ -42,12 +42,18 @@ Manages the replication agreements of an IPA server. >>> \fBforce\-sync\fR >>> \- Immediately flush any data to be replicated from a server specified with >>> the \-\-from option >>> .TP >>> +\fBcleanup\fR >>> +\- Remove leftover references to a deleted master. >>> +.TP >>> >>> >>> The cleanup procedure was implemented rather as an option to the del command >>> than a separate command. >>> >>> >>> 5) My favorite - new cleanup procedure user prompt miss the --force option >>> useful for test automation >>> >>> + if not ipautil.user_input("Continue to clean master?", False): >>> + sys.exit("Cleanup aborted") >>> >>> >>> Otherwise the patch looks good. >>> >>> Martin >>> >> >> I pulled the abort-ruv stuff out. It was just easier to stuff it in here than >> rebasing back, but yeah, its cleaner this way. >> >> No need to check force for clean because it is already required (can't contact >> the deleted master, it's gone). >> >> rob > > The patch works fine now. So ACK when the dependent patch 1031 is pushed. > > Martin > Pushed to master, ipa-3-0. Martin From jfenal at gmail.com Mon Sep 17 17:59:05 2012 From: jfenal at gmail.com (=?UTF-8?B?SsOpcsO0bWUgRmVuYWw=?=) Date: Mon, 17 Sep 2012 19:59:05 +0200 Subject: [Freeipa-devel] [PATCH] 0079 Update the pot file (translation source) In-Reply-To: <5056DCD9.9020805@redhat.com> References: <50504AEF.9090209@redhat.com> <5052322F.2040009@redhat.com> <50531A48.9080604@redhat.com> <5056DCD9.9020805@redhat.com> Message-ID: 2012/9/17 Petr Viktorin > On 09/14/2012 09:36 PM, J?r?me Fenal wrote: > >> 2012/9/14 Petr Viktorin > >> >> > > I pushed the pot manually. >> Since we have infrequent explicit string freezes I don't think it's >> necessary to configure automatic pot updates again. >> >> >> Thanks Petr! >> >> Actually, having the strings updated on Transifex on a regular basis >> makes it (IMHO) more manageable for translators to update the >> translations even before a string freeze. Translating a dozen strings >> per week is lighter than a mere 339 strings. >> > > A possible problem with this approach is that the translators would see > and translate messages that don't make it into the final version. Do you > think a more even workload would be worth the occasional extra work? > Having some extra work from time to time is OK. Having a huge batch of strings to translate on a deadline is uneasy. Especially with day job ramping up... ;-) > I would like to change our i18n workflow/infrastructure. I was planning to > (re-)start discussing this after the 3.0 release rush is done. It should be > possible to do what you suggest. > Cool, thanks. > I also don't know if pulls from Transifex or push from your side has an >> effect of keeping memory (in suggestions) of past or close enough >> strings from the past for small modifications. >> > > Sadly, I don't know much about Transifex itself. Perhaps ask the team > there, and request the feature if it's missing. > This item is not that important, more a wish. > Another comment/request, I don't know given my zero-level Python-fu: >> would it be possible to break down the huge __doc__ strings in plugins >> into smaller parts, as a small modification would impact a smaller >> strings, easing maintenance instead of trying to track the one character >> modification in a 2000 chars text. >> >> Does Python support concatenations of __doc___ strings? >> > > That is be possible on the Python side. I'm not sure how Transifex (and > other translation tools) would cope with text split between several > messages -- sorting and filtering the messages could take things out of > context. > I'm aware of this issue, translators for others languages may raise their hands. I could also probe trans at fp.o on that matter, to reach all of them. That will nevertheless make changes more atomic, and overall easier to manage on the longer term. In the past, changes were just a matter of adding one paragraph, or adding one more usage example. Or fixing a typo. Which is way harder to spot when you have a 1000 chars strings tagged as changed. Regards, J. -- J?r?me Fenal - jfenal AT gmail.com - http://fenal.org/ Paris.pm - http://paris.mongueurs.net/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rcritten at redhat.com Mon Sep 17 18:10:50 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 17 Sep 2012 14:10:50 -0400 Subject: [Freeipa-devel] [PATCH] 0077 Check direct/reverse hostname/address resolution in ipa-replica-install In-Reply-To: <50570D68.9040501@redhat.com> References: <5046009E.2020306@redhat.com> <50463DE6.9070300@redhat.com> <5046FF0E.1000504@redhat.com> <504FA7A3.5080201@redhat.com> <5050B1ED.70307@redhat.com> <50524387.4020800@redhat.com> <5052D2CC.2000405@redhat.com> <50570D68.9040501@redhat.com> Message-ID: <505767AA.5080607@redhat.com> Petr Viktorin wrote: > On 09/14/2012 08:46 AM, Martin Kosek wrote: >> On 09/13/2012 10:35 PM, Rob Crittenden wrote: >>> Petr Viktorin wrote: >>>> On 09/11/2012 11:05 PM, Rob Crittenden wrote: >>>>> Petr Viktorin wrote: >>>>>> On 09/04/2012 07:44 PM, Rob Crittenden wrote: >>>>>>> Petr Viktorin wrote: >>>>>>>> >>>>>>>> https://fedorahosted.org/freeipa/ticket/2845 >>>>>>> >>>>>>> Shouldn't this also call verify_fqdn() on the local hostname and not >>>>>>> just the master? I think this would eventually fail in the conncheck >>>>>>> but >>>>>>> what if that was skipped? >>>>>>> >>>>>>> rob >>>>>> >>>>>> A few lines above there is a call to get_host_name, which will call >>>>>> verify_fqdn. >>>>>> >>>>> >>>>> I double-checked this, it fails in conncheck. Here are my steps: >>>>> >>>>> # ipa-server-install --setup-dns >>>>> # ipa-replica-prepare replica.example.com --ip-address=192.168.100.2 >>>>> # ipa host-del replica.example.com >>>>> >>>>> On replica, set DNS to IPA master, with hostname in /etc/hosts. >>>>> >>>>> # ipa-replica-install ... >>>>> >>>>> The verify_fqdn() passes because the resolver uses /etc/hosts. >>>>> >>>>> The conncheck fails: >>>>> >>>>> Execute check on remote master >>>>> Check connection from master to remote replica 'replica.example.com': >>>>> >>>>> Remote master check failed with following error message(s): >>>>> Could not chdir to home directory /home/admin: No such file or >>>>> directory >>>>> Port check failed! Unable to resolve host name 'replica.example.com' >>>>> >>>>> Connection check failed! >>>>> Please fix your network settings according to error messages above. >>>>> If the check results are not valid it can be skipped with >>>>> --skip-conncheck parameter. >>>>> >>>>> The DNS test happens much further after this, and I get why, I just >>>>> don't see how useful it is unless the --skip-conncheck is used. >>>> >>>> For the record, it's because we need to check if the host has DNS >>>> installed. We need a LDAP connection to check this. >>>> >>>>> ipa-replica-install ~rcrit/replica-info-replica.example.com.gpg >>>>> --skip-conncheck >>>>> Directory Manager (existing master) password: >>>>> >>>>> ipa : ERROR Could not resolve hostname replica.example.com >>>>> using DNS. Clients may not function properly. Please check your DNS >>>>> setup. (Note that this check queries IPA DNS directly and ignores >>>>> /etc/hosts.) >>>>> Continue? [no]: >>>>> >>>>> So I guess, what are the intentions here? It is certainly better than >>>>> before. >>>>> >>>>> rob >>>> >>>> If the replica is in the master's /etc/hosts, but not in DNS, the >>>> conncheck will succeed. This check explicitly queries IPA records only >>>> and ignores /etc/hosts so it'll notice this case and warn. >>>> >>> >>> Ok, like I said, this is better than we have. Just one nit then you >>> get an ack: >>> >>> + # If remote host has DNS, check forward/reverse resolution >>> + try: >>> + entry = conn.find_entries(u'cn=dns', >>> base_dn=DN(api.env.basedn)) >>> + except errors.NotFound: >>> >>> u'cn=dns' should be str(constants.container_dns). >>> >>> rob >> >> This is a search filter, Petr could use the one I already have in >> "dns.py::get_dns_masters()" function: >> '(&(objectClass=ipaConfigObject)(cn=DNS))' >> >> For performance sake, I would also not search in the entire tree, but >> limit the >> search only to: >> >> DN(('cn', 'masters'), ('cn', 'ipa'), ('cn', 'etc'), api.env.basedn) >> >> Martin >> > > Attaching updated patch with Martin's suggestions. I think what Martin had in mind was: if api.Object.dnsrecord.get_dns_masters(): ... From rcritten at redhat.com Mon Sep 17 19:35:35 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 17 Sep 2012 15:35:35 -0400 Subject: [Freeipa-devel] [PATCH] 302 Stricter IP network validator in dnszone-add command In-Reply-To: <5051D666.1060604@redhat.com> References: <5047077E.1040000@redhat.com> <50471BBD.1070506@redhat.com> <504727E5.5040101@redhat.com> <50472B1C.2070207@redhat.com> <50472DF6.8000207@redhat.com> <5047314B.30201@redhat.com> <5051D666.1060604@redhat.com> Message-ID: <50577B87.3080204@redhat.com> Martin Kosek wrote: > On 09/05/2012 01:02 PM, Jan Cholasta wrote: >> Dne 5.9.2012 12:48, Martin Kosek napsal(a): >>> On 09/05/2012 12:36 PM, Jan Cholasta wrote: >>>> Dne 5.9.2012 12:22, Petr Spacek napsal(a): >>>>> On 09/05/2012 11:30 AM, Jan Cholasta wrote: >>>>>> Dne 5.9.2012 10:04, Martin Kosek napsal(a): >>>>>>> We allowed IP addresses without network specification which lead >>>>>>> to unexpected results when the zone was being created. We should rather >>>>>>> strictly require the prefix/netmask specifying the IP network that >>>>>>> the reverse zone should be created for. This is already done in >>>>>>> Web UI. >>>>>>> >>>>>>> A unit test exercising this new validation was added. >>>>>>> >>>>>>> https://fedorahosted.org/freeipa/ticket/2461 >>>>>>> >>>>>> >>>>>> I don't like this much. I would suggest using CheckedIPAddress and not >>>>>> forcing >>>>>> the user to enter the prefix length instead. >>>>>> >>>>>> CheckedIPAddress uses a sensible default prefix length if one is not >>>>>> specified >>>>>> (class-based for IPv4, /64 for IPv6) as opposed to IPNetwork (/32 for >>>>>> IPv4, >>>>>> /128 for IPv6 - this causes the erroneous reverse zones to be created as >>>>>> described in the ticket). >>>>>> >>>>> Hello, >>>>> >>>>> I don't like automatic netmask guessing. I have met class-based guessing >>>>> in Windows (XP?) and I was forced to overwrite default mask all the time >>>>> ... >>>> >>>> If there was no guessing, you would have to write the netmask anyway, so I >>>> don't see any harm in guessing here. >>>> >>>>> >>>>> IMHO there is no "sensible default prefix" in real world. I sitting on >>>>> network with /23 prefix right now. Also, I have never seen 10.x network >>>>> with /8 prefix. >>>>> >>>> >>>> While this might be true for IPv4 in some cases, /64 is perfectly sensible for >>>> IPv6. Also, I have never seen 192.168.x.x network with non-/24 prefix. >>>> >>>> Honza >>>> >>> >>> While this may be true for 192.168.x.x, it does not apply for 10.x.x.x networks >>> as Petr already pointed out. I don't think that there will be many people >>> expecting that a reverse zone of 10.0.0.0/24 would be created. >> >> And they would be correct, because the default prefix length for a class A >> network is /8, not /24. >> >>> >>> And since FreeIPA is mainly deployed to internal networks, I assume this will >>> be the case of most users. >>> >>> Martin >>> >> >> OK, but what about IPv6? Correct me if I'm wrong, but the prefix length is >> going to be /64 99% of the time for IPv6. >> >> The installer uses /24 for IPv4 addresses and /64 for IPv6 addresses, maybe >> this should be used as a default here as well. >> >> Honza >> > > In the end, I choose a more liberal approach and instead of defining a more > stricter validator for IPv4 only I rather used approach already implemented in > the installers, i.e. default length of network prefix is 24 for IPv4 and 64 for > IPv6. > > Updated patch attached. > > Martin Works for me. I wonder if this is a candidate for some more unit tests... rob From edewata at redhat.com Tue Sep 18 02:45:26 2012 From: edewata at redhat.com (Endi Sukma Dewata) Date: Mon, 17 Sep 2012 21:45:26 -0500 Subject: [Freeipa-devel] [PATCH] 212 Fix integer validation when boundary value is empty string In-Reply-To: <504F542A.7060204@redhat.com> References: <504F542A.7060204@redhat.com> Message-ID: <5057E046.3030303@redhat.com> On 9/11/2012 10:09 AM, Petr Vobornik wrote: > There was an error in number validation check. If boundary value was an > empty string, validation of a number always failed. This patch fixes the > problem by not performing the check in these cases. > > Basic unit tests for IPA.metadata_validator created. ACK. Some comments: 1. Instead of IPA.not_defined() it might be better to call it IPA.defined() to avoid double negations like this: if (!IPA.not_defined(metadata.minvalue, true) ... 2. The check_empty_str probably could made optional and the value would be true by default. It will make the code cleaner. -- Endi S. Dewata From edewata at redhat.com Tue Sep 18 02:45:32 2012 From: edewata at redhat.com (Endi Sukma Dewata) Date: Mon, 17 Sep 2012 21:45:32 -0500 Subject: [Freeipa-devel] [PATCH] 214 Fix jquery error when using '??' in a pkey In-Reply-To: <50532A8B.2060309@redhat.com> References: <50532A8B.2060309@redhat.com> Message-ID: <5057E04C.6000809@redhat.com> On 9/14/2012 8:00 AM, Petr Vobornik wrote: > This patch is only for FreeIPA 2.2. It is already fixed in 3.0. > > If '??' is used in a adder dialog as a pkey it can cause > "jQuery15208158273949015573_1346241267446 was not called" error. > > Update of jquery library fixes the issue. Update reveals an incorrect > handler definition issue in ssh_key_widget, which is also fixed. > > https://bugzilla.redhat.com/show_bug.cgi?id=855278 > https://fedorahosted.org/freeipa/ticket/3073 ACK. -- Endi S. Dewata From mkosek at redhat.com Tue Sep 18 06:50:58 2012 From: mkosek at redhat.com (Martin Kosek) Date: Tue, 18 Sep 2012 08:50:58 +0200 Subject: [Freeipa-devel] [PATCH] Fix various typos Message-ID: <505819D2.8030700@redhat.com> ACK for typo fixes made by Yuri Chornoivan (patch attached). Pushed to master, ipa-3-0. Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-ychornoivan-typos.patch Type: text/x-patch Size: 18906 bytes Desc: not available URL: From mkosek at redhat.com Tue Sep 18 09:14:31 2012 From: mkosek at redhat.com (Martin Kosek) Date: Tue, 18 Sep 2012 11:14:31 +0200 Subject: [Freeipa-devel] [PATCH] 310 Properly convert DN in ipa-client-install Message-ID: <50583B77.4030807@redhat.com> Pushed to master, ipa-3-0 as a one-liner. --- ipa-client-install crashed when IPA server anonymous access was disabled and base DN was thus generated via realm_to_suffix function which, however, returns a DN object and not string. DN was converted to string, ipa-client-install no longer crashes in this scenario. https://fedorahosted.org/freeipa/ticket/3088 -- Martin Kosek Senior Software Engineer - Identity Management Team Red Hat Inc. -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-310-properly-convert-dn-in-ipa-client-install.patch Type: text/x-patch Size: 1312 bytes Desc: not available URL: From tbabej at redhat.com Tue Sep 18 09:21:11 2012 From: tbabej at redhat.com (Tomas Babej) Date: Tue, 18 Sep 2012 11:21:11 +0200 Subject: [Freeipa-devel] [PATCH 0006] Improves sssd.conf handling during ipa-client uninstall In-Reply-To: <5050AA68.2010300@redhat.com> References: <717623750.1969105.1345208663118.JavaMail.root@redhat.com> <503B5D09.7040401@redhat.com> <503B6EC8.3060508@redhat.com> <20120827133727.GG8612@zeppelin.brq.redhat.com> <503B8A65.30100@redhat.com> <503E1114.3080309@redhat.com> <5050AA68.2010300@redhat.com> Message-ID: <50583D07.6040001@redhat.com> On 09/12/2012 05:29 PM, Martin Kosek wrote: > On 08/29/2012 02:54 PM, Tomas Babej wrote: >> On 08/27/2012 04:55 PM, Martin Kosek wrote: >>> On 08/27/2012 03:37 PM, Jakub Hrozek wrote: >>>> On Mon, Aug 27, 2012 at 02:57:44PM +0200, Martin Kosek wrote: >>>>> I think that the right behavior of SSSD conf uninstall should be the >>>>> following: >>>>> >>>>> * sssd.conf existed before IPA install + non-IPA domains in sssd.conf found: >>>>> - move backed conf up sssd.conf.bkp (and inform the user) >>>>> - use SSSDConfig delete_domain function to remove ipa domain from sssd.conf >>>>> - restart sssd afterwards >>>> I'm confused here, which of the files is the original >>>> pre-ipa-client-install file? >>> This is the "backed up sssd.conf". I thought that it may be useful for user to >>> still have an access to it after uninstall. >>> >>>> How does the non-ipa domain end up in the sssd.conf file? Does it have >>>> to be configured manually or does ipa-client-install merge the list of >>>> domains on installation? >>> ipa-client-install merge the list of the domains. It overrides the old >>> sssd.conf only when we cannot parse the sssd.conf and --preserve-sssd option >>> was not set. >>> >>> Martin >> Hi, >> >> The sssd.conf file is no longer left behind in case sssd was not >> configured before the installation. However, the patch goes behind >> the scope of this ticked and improves the handling of sssd.conf >> during the ipa-client-install --uninstall in general. >> >> The current behaviour (well documented in source code) is as follows: >> - In general, the IPA domain is simply removed from the sssd.conf >> file, instead of sssd.conf being rewritten from the backup. This >> preserves any domains added after installation. >> >> - If sssd.conf existed before the installation, it is restored to >> sssd.conf.bkp. However, any IPA domains from pre-installation >> sssd.conf should have been merged during the installation. >> >> - If sssd.conf did not exist before the installation, and no other >> domains than IPA domain exist in it, the patch makes sure that >> sssd.conf is moved to sssd.conf.deleted so user experiences no >> crash during any next installation due to its existence. >> >> https://fedorahosted.org/freeipa/ticket/2740 >> >> Tomas >> > Good job, SSSD uninstall process now looks more consistent and better > documented. I just found the following (mainly minor) issues. Comments in the > patch: > > diff --git a/ipa-client/ipa-install/ipa-client-install > b/ipa-client/ipa-install/ipa-client-install > index > 2e65921e8de2dfe68443f5b5875954d71dd48ed2..c5cef15e1fb3a3e1d7cfd070f4288d3839accfc8 > 100755 > --- a/ipa-client/ipa-install/ipa-client-install > +++ b/ipa-client/ipa-install/ipa-client-install > @@ -183,6 +183,36 @@ def nssldap_exists(): > > return (retval, files_found) > > +# helper function for uninstall > +# deletes IPA domain from sssd.conf > +def delete_IPA_domain(): > > Function names should be lowercase -> delete_ipa_domain > > + sssd = ipaservices.service('sssd') > + try: > + sssdconfig = SSSDConfig.SSSDConfig() > + sssdconfig.import_config() > + domains = sssdconfig.list_active_domains() > + > + IPA_domain_name = None > > Variables should be lowercase -> ipa_domain_name > > + > + for name in domains: > + domain = sssdconfig.get_domain(name) > + try: > + provider = domain.get_option('id_provider') > + if provider == "ipa": > + IPA_domain_name = name > + break > + except SSSDConfig.NoOptionError: > + continue > + > + if IPA_domain_name != None: > > Do not use != with None, True, False - use "is not None". > > + sssdconfig.delete_domain(IPA_domain_name) > + sssdconfig.write() > + else: > + root_logger.warning("IPA domain could not be found in " + > + "sssd.conf and therefore not deleted") > + except IOError: > + root_logger.warning("IPA domain could not be deleted. No access to the > sssd.conf file.") > > There should be full path to sssd.conf in this error message. It is very useful > sometimes. > > + > def uninstall(options, env): > > if not fstore.has_files(): > @@ -212,7 +242,12 @@ def uninstall(options, env): > sssdconfig = SSSDConfig.SSSDConfig() > sssdconfig.import_config() > domains = sssdconfig.list_active_domains() > - if len(domains) > 1: > + all_domains = sssdconfig.list_domains() > + > + # we consider all the domains, because handling sssd.conf > + # during uninstall is dependant on was_sssd_configured flag > + # so the user does not lose info about inactive domains > + if len(all_domains) > 1: > # There was more than IPA domain configured > was_sssd_configured = True > for name in domains: > @@ -349,6 +384,62 @@ def uninstall(options, env): > "Failed to remove krb5/LDAP configuration: %s", str(e)) > return CLIENT_INSTALL_ERROR > > + # Next if-elif-elif construction deals with sssd.conf file. > + # Old pre-IPA domains are preserved due merging the old sssd.conf > + # during the installation of ipa-client but any new domains are > + # only present in sssd.conf now, so we don't want to delete them > + # by rewriting sssd.conf file. IPA domain is removed gracefully. > + > + # SSSD was installed before our installation and other non-IPA domains > + # found, restore backed up sssd.conf to sssd.conf.bkp and remove IPA > + # domain from the current sssd.conf > + if was_sssd_installed and was_sssd_configured: > + root_logger.info( > + "The original configuration of SSSD included other domains than " + > + "the IPA-based one.") > + > + delete_IPA_domain() > + > > + try: > + os.rename("/etc/sssd/sssd.conf","/etc/sssd/sssd.conf.current") > + if fstore.restore_file("/etc/sssd/sssd.conf"): > + os.rename("/etc/sssd/sssd.conf","/etc/sssd/sssd.conf.bkp") > + os.rename("/etc/sssd/sssd.conf.current","/etc/sssd/sssd.conf") > > I don't like this very much. I would rather not mess with /etc/sssd/sssd.conf, > it may surprise other tools/user and we could also end with no > /etc/sssd/sssd.conf at all if OSError is raised in second or third step. > > I would rather enhance fstore and implement function like "restore_file_to" > which would accept a second parameter with a custom path, like > "/etc/sssd/sssd.conf.bkp". > > + except OSError: > + pass > > The exception should be at least logged to debug log. > > + > + root_logger.info("Original pre-IPA SSSD configuration file was " + > + "restored to /etc/sssd/sssd.conf.bkp.") > + root_logger.info("IPA domain removed from current one, " + > + "restarting SSSD service") > + sssd = ipaservices.service('sssd') > + try: > + sssd.restart() > + except CalledProcessError: > + root_logger.warning("SSSD service restart was unsuccessful.") > + > + # SSSD was not installed before our installation, but other domains found, > + # delete IPA domain, but leave other domains intact > + elif not was_sssd_installed and was_sssd_configured: > + delete_IPA_domain() > + root_logger.info("Other domains than IPA domain found, " + > + "IPA domain was removed from sssd.conf.") > + try: > + sssd.restart() > + except CalledProcessError: > + root_logger.warning("SSSD service restart was unsuccessful.") > + > + # SSSD was not installed before our installation, and no other domains > + # than IPA are configured in sssd.conf - make sure config file is removed > + elif not was_sssd_installed and not was_sssd_configured: > + try: > + os.rename("/etc/sssd/sssd.conf","/etc/sssd/sssd.conf.deleted") > + except OSError: > + pass > > > Error should be logged to debug log. > > + > + root_logger.info("Redundant SSSD configuration file " + > + "/etc/sssd/sssd.conf was moved to /etc/sssd.conf.deleted") > + > if fstore.has_files(): > root_logger.info("Restoring client configuration files") > fstore.restore_all_files() > @@ -428,20 +519,6 @@ def uninstall(options, env): > if was_sshd_configured and ipaservices.knownservices.sshd.is_running(): > ipaservices.knownservices.sshd.restart() > > - if was_sssd_installed and was_sssd_configured: > - # SSSD was installed before our installation, config now is restored, > restart it > - root_logger.info( > - "The original configuration of SSSD included other domains than " + > - "the IPA-based one.") > - root_logger.info( > - "Original configuration file was restored, restarting SSSD " + > - "service.") > - sssd = ipaservices.service('sssd') > - try: > - sssd.restart() > - except CalledProcessError: > - root_logger.warning("SSSD service restart was unsuccessful.") > - > if not options.unattended: > root_logger.info( > "The original nsswitch.conf configuration has been restored.") Final (hopefully) version attached. Tomas -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-tbabej-0006-3-Improves-sssd.conf-handling-during-ipa-client-uninst.patch Type: text/x-patch Size: 9019 bytes Desc: not available URL: From sbose at redhat.com Tue Sep 18 10:42:49 2012 From: sbose at redhat.com (Sumit Bose) Date: Tue, 18 Sep 2012 12:42:49 +0200 Subject: [Freeipa-devel] [PATCH] 0073 Add trust verification code In-Reply-To: <20120917154436.GA12295@redhat.com> References: <20120917154436.GA12295@redhat.com> Message-ID: <20120918104249.GN28400@localhost.localdomain> On Mon, Sep 17, 2012 at 06:44:36PM +0300, Alexander Bokovoy wrote: > Hi, > > Following patch adds trust verification sequence to the case when we > establish trust with knowledge of AD administrative credentials. > > As we found out, in order to validate/verify trust, one has to have > administrative credentials for the trusted domain, since there are > few RPCs that should be performed against trusted domain's DC's LSA > and NetLogon pipes and these are protected by administrative credentials. > > Thus, when we know admin credentials for the remote domain, we can > perform the trust validation. > > https://fedorahosted.org/freeipa/ticket/2763 > Just a short feedback. The patch is working as expected, for a newly created trust Windows will send a TGS request to the IPA KDC without explicit validation on the windows side. Currently I have some issues in my test setup so that I can not give a full ACK atm. bye, Sumit > > -- > / Alexander Bokovoy From pviktori at redhat.com Tue Sep 18 11:17:06 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Tue, 18 Sep 2012 13:17:06 +0200 Subject: [Freeipa-devel] [PATCH] 0077 Check direct/reverse hostname/address resolution in ipa-replica-install In-Reply-To: <505767AA.5080607@redhat.com> References: <5046009E.2020306@redhat.com> <50463DE6.9070300@redhat.com> <5046FF0E.1000504@redhat.com> <504FA7A3.5080201@redhat.com> <5050B1ED.70307@redhat.com> <50524387.4020800@redhat.com> <5052D2CC.2000405@redhat.com> <50570D68.9040501@redhat.com> <505767AA.5080607@redhat.com> Message-ID: <50585832.8060005@redhat.com> On 09/17/2012 08:10 PM, Rob Crittenden wrote: > Petr Viktorin wrote: >> On 09/14/2012 08:46 AM, Martin Kosek wrote: >>> On 09/13/2012 10:35 PM, Rob Crittenden wrote: >>>> Petr Viktorin wrote: >>>>> On 09/11/2012 11:05 PM, Rob Crittenden wrote: >>>>>> Petr Viktorin wrote: >>>>>>> On 09/04/2012 07:44 PM, Rob Crittenden wrote: >>>>>>>> Petr Viktorin wrote: >>>>>>>>> >>>>>>>>> https://fedorahosted.org/freeipa/ticket/2845 >>>>>>>> >>>>>>>> Shouldn't this also call verify_fqdn() on the local hostname and >>>>>>>> not >>>>>>>> just the master? I think this would eventually fail in the >>>>>>>> conncheck >>>>>>>> but >>>>>>>> what if that was skipped? >>>>>>>> >>>>>>>> rob >>>>>>> >>>>>>> A few lines above there is a call to get_host_name, which will call >>>>>>> verify_fqdn. >>>>>>> >>>>>> >>>>>> I double-checked this, it fails in conncheck. Here are my steps: >>>>>> >>>>>> # ipa-server-install --setup-dns >>>>>> # ipa-replica-prepare replica.example.com --ip-address=192.168.100.2 >>>>>> # ipa host-del replica.example.com >>>>>> >>>>>> On replica, set DNS to IPA master, with hostname in /etc/hosts. >>>>>> >>>>>> # ipa-replica-install ... >>>>>> >>>>>> The verify_fqdn() passes because the resolver uses /etc/hosts. >>>>>> >>>>>> The conncheck fails: >>>>>> >>>>>> Execute check on remote master >>>>>> Check connection from master to remote replica 'replica.example.com': >>>>>> >>>>>> Remote master check failed with following error message(s): >>>>>> Could not chdir to home directory /home/admin: No such file or >>>>>> directory >>>>>> Port check failed! Unable to resolve host name 'replica.example.com' >>>>>> >>>>>> Connection check failed! >>>>>> Please fix your network settings according to error messages above. >>>>>> If the check results are not valid it can be skipped with >>>>>> --skip-conncheck parameter. >>>>>> >>>>>> The DNS test happens much further after this, and I get why, I just >>>>>> don't see how useful it is unless the --skip-conncheck is used. >>>>> >>>>> For the record, it's because we need to check if the host has DNS >>>>> installed. We need a LDAP connection to check this. >>>>> >>>>>> ipa-replica-install ~rcrit/replica-info-replica.example.com.gpg >>>>>> --skip-conncheck >>>>>> Directory Manager (existing master) password: >>>>>> >>>>>> ipa : ERROR Could not resolve hostname replica.example.com >>>>>> using DNS. Clients may not function properly. Please check your DNS >>>>>> setup. (Note that this check queries IPA DNS directly and ignores >>>>>> /etc/hosts.) >>>>>> Continue? [no]: >>>>>> >>>>>> So I guess, what are the intentions here? It is certainly better than >>>>>> before. >>>>>> >>>>>> rob >>>>> >>>>> If the replica is in the master's /etc/hosts, but not in DNS, the >>>>> conncheck will succeed. This check explicitly queries IPA records only >>>>> and ignores /etc/hosts so it'll notice this case and warn. >>>>> >>>> >>>> Ok, like I said, this is better than we have. Just one nit then you >>>> get an ack: >>>> >>>> + # If remote host has DNS, check forward/reverse resolution >>>> + try: >>>> + entry = conn.find_entries(u'cn=dns', >>>> base_dn=DN(api.env.basedn)) >>>> + except errors.NotFound: >>>> >>>> u'cn=dns' should be str(constants.container_dns). >>>> >>>> rob >>> >>> This is a search filter, Petr could use the one I already have in >>> "dns.py::get_dns_masters()" function: >>> '(&(objectClass=ipaConfigObject)(cn=DNS))' >>> >>> For performance sake, I would also not search in the entire tree, but >>> limit the >>> search only to: >>> >>> DN(('cn', 'masters'), ('cn', 'ipa'), ('cn', 'etc'), api.env.basedn) >>> >>> Martin >>> >> >> Attaching updated patch with Martin's suggestions. > > I think what Martin had in mind was: > > if api.Object.dnsrecord.get_dns_masters(): > ... > I didn't want to do this because api.Object.* use our global ldap2 Backend, which is hardwired to query localhost. I see now that I can hack around this, and we already do this in ipa-replica-install. I've extracted the hack and reused it to get the DNS masters. -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0077-03-Check-direct-reverse-hostname-address-resolution-in-.patch Type: text/x-patch Size: 9996 bytes Desc: not available URL: From pvoborni at redhat.com Tue Sep 18 11:36:13 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Tue, 18 Sep 2012 13:36:13 +0200 Subject: [Freeipa-devel] [PATCH] 212 Fix integer validation when boundary value is empty string In-Reply-To: <5057E046.3030303@redhat.com> References: <504F542A.7060204@redhat.com> <5057E046.3030303@redhat.com> Message-ID: <50585CAD.5030905@redhat.com> On 09/18/2012 04:45 AM, Endi Sukma Dewata wrote: > On 9/11/2012 10:09 AM, Petr Vobornik wrote: >> There was an error in number validation check. If boundary value was an >> empty string, validation of a number always failed. This patch fixes the >> problem by not performing the check in these cases. >> >> Basic unit tests for IPA.metadata_validator created. > > ACK. Some comments: Updated patch attached. > > 1. Instead of IPA.not_defined() it might be better to call it > IPA.defined() to avoid double negations like this: > > if (!IPA.not_defined(metadata.minvalue, true) ... > Function renamed, logic negated. Unit tests for IPA.defined added. > 2. The check_empty_str probably could made optional and the value would > be true by default. It will make the code cleaner. > I wanted the function to be more general - used somewhere else in a future. In general, I consider empty string as a defined value so for me 'false' is a good default value of the argument. I agree that in this case it would look cleaner. -- Petr Vobornik -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pvoborni-0212-1-Fix-integer-validation-when-boundary-value-is-empty-.patch Type: text/x-patch Size: 9966 bytes Desc: not available URL: From pvoborni at redhat.com Tue Sep 18 11:40:57 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Tue, 18 Sep 2012 13:40:57 +0200 Subject: [Freeipa-devel] [PATCH] 214 Fix jquery error when using '??' in a pkey In-Reply-To: <5057E04C.6000809@redhat.com> References: <50532A8B.2060309@redhat.com> <5057E04C.6000809@redhat.com> Message-ID: <50585DC9.3050300@redhat.com> On 09/18/2012 04:45 AM, Endi Sukma Dewata wrote: > On 9/14/2012 8:00 AM, Petr Vobornik wrote: >> This patch is only for FreeIPA 2.2. It is already fixed in 3.0. >> >> If '??' is used in a adder dialog as a pkey it can cause >> "jQuery15208158273949015573_1346241267446 was not called" error. >> >> Update of jquery library fixes the issue. Update reveals an incorrect >> handler definition issue in ssh_key_widget, which is also fixed. >> >> https://bugzilla.redhat.com/show_bug.cgi?id=855278 >> https://fedorahosted.org/freeipa/ticket/3073 > > ACK. > Pushed to ipa-2-2 -- Petr Vobornik From pviktori at redhat.com Tue Sep 18 12:03:57 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Tue, 18 Sep 2012 14:03:57 +0200 Subject: [Freeipa-devel] [PATCH] 0079 Update the pot file (translation source) In-Reply-To: References: <50504AEF.9090209@redhat.com> <5052322F.2040009@redhat.com> <50531A48.9080604@redhat.com> <5056DCD9.9020805@redhat.com> Message-ID: <5058632D.5020307@redhat.com> On 09/17/2012 07:59 PM, J?r?me Fenal wrote: > 2012/9/17 Petr Viktorin > > > On 09/14/2012 09:36 PM, J?r?me Fenal wrote: > > 2012/9/14 Petr Viktorin >> > > > I pushed the pot manually. > Since we have infrequent explicit string freezes I don't > think it's > necessary to configure automatic pot updates again. > > > Thanks Petr! > > Actually, having the strings updated on Transifex on a regular basis > makes it (IMHO) more manageable for translators to update the > translations even before a string freeze. Translating a dozen > strings > per week is lighter than a mere 339 strings. > > > A possible problem with this approach is that the translators would > see and translate messages that don't make it into the final > version. Do you think a more even workload would be worth the > occasional extra work? > > > Having some extra work from time to time is OK. Having a huge batch of > strings to translate on a deadline is uneasy. Especially with day job > ramping up... ;-) > > I would like to change our i18n workflow/infrastructure. I was > planning to (re-)start discussing this after the 3.0 release rush is > done. It should be possible to do what you suggest. > > > Cool, thanks. > > I also don't know if pulls from Transifex or push from your side > has an > effect of keeping memory (in suggestions) of past or close enough > strings from the past for small modifications. > > > Sadly, I don't know much about Transifex itself. Perhaps ask the > team there, and request the feature if it's missing. > > > This item is not that important, more a wish. > > Another comment/request, I don't know given my zero-level Python-fu: > would it be possible to break down the huge __doc__ strings in > plugins > into smaller parts, as a small modification would impact a smaller > strings, easing maintenance instead of trying to track the one > character > modification in a 2000 chars text. > > Does Python support concatenations of __doc___ strings? > > > That is be possible on the Python side. I'm not sure how Transifex > (and other translation tools) would cope with text split between > several messages -- sorting and filtering the messages could take > things out of context. > > > I'm aware of this issue, translators for others languages may raise > their hands. I could also probe trans at fp.o on that matter, to reach all > of them. If the answer is not in a FAQ somewhere, that would be nice. I have a feeling that the communication between developers and translators could be better. It seems like many translators think they're given the strings on a silver platter and have to deal with what's there, and the developers don't really know the translators' needs. Thanks for reaching out! We want to help (though changes may be slow, and won't happen during a string freeze), and definitely appreciate these comments so we can better plan for the future. > That will nevertheless make changes more atomic, and overall easier to > manage on the longer term. In the past, changes were just a matter of > adding one paragraph, or adding one more usage example. Or fixing a > typo. Which is way harder to spot when you have a 1000 chars strings > tagged as changed. Ideally Transifex would show the difference between the old and the new. Unfortunately the data's not there to do this automatically :( For typos, there should be a way to leave the old translations unchanged (since the translations won't have the typo). Gettext has the "fuzzy" feature for this, but it has problems so it's turned off for IPA. I'll quote John here (apologies for posting private conversation): > Never use fuzzy entries. The logic used to select a fuzzy entry is > poor, it often picks up entirely wrong translations, they are evil. > We've had way too many problems with message catalogs getting > corrupted due to the introduction of fuzzy entries. Given that fuzzy > entries are so seldom correct and are mostly of value to translators > and that translators have a lazy habit of just accepting the > incorrect fuzzy suggestion and promoting to a full translation they > should not be used at all. I've spent days in the past trying to > unwind corrupted files due to the incorrect application of fuzzy > translations. I really don't want to do that again :-) Gettext was designed for delayed communication -- the translators would download the file, work for a few weeks, then mail the result back. There could be various versions of POT and PO files floating around. The metadata and logic, including the fuzzy system, was built around that. With Transifex, there's one version of everything, and we have access to the translations as they're written. There might be ways to exploit this to make things better. > Regards, > > J. -- Petr? From sbose at redhat.com Tue Sep 18 12:15:39 2012 From: sbose at redhat.com (Sumit Bose) Date: Tue, 18 Sep 2012 14:15:39 +0200 Subject: [Freeipa-devel] [PATCH] 0073 Add trust verification code In-Reply-To: <20120918104249.GN28400@localhost.localdomain> References: <20120917154436.GA12295@redhat.com> <20120918104249.GN28400@localhost.localdomain> Message-ID: <20120918121539.GO28400@localhost.localdomain> On Tue, Sep 18, 2012 at 12:42:49PM +0200, Sumit Bose wrote: > On Mon, Sep 17, 2012 at 06:44:36PM +0300, Alexander Bokovoy wrote: > > Hi, > > > > Following patch adds trust verification sequence to the case when we > > establish trust with knowledge of AD administrative credentials. > > > > As we found out, in order to validate/verify trust, one has to have > > administrative credentials for the trusted domain, since there are > > few RPCs that should be performed against trusted domain's DC's LSA > > and NetLogon pipes and these are protected by administrative credentials. > > > > Thus, when we know admin credentials for the remote domain, we can > > perform the trust validation. > > > > https://fedorahosted.org/freeipa/ticket/2763 > > > > Just a short feedback. The patch is working as expected, for a newly > created trust Windows will send a TGS request to the IPA KDC without > explicit validation on the windows side. Currently I have some issues > in my test setup so that I can not give a full ACK atm. > ok, ACK. Nevertheless it would be nice if Petr can check for any implications to the web UI with respect to the status of the trust. bye, Sumit > bye, > Sumit > > > > > -- > > / Alexander Bokovoy > > _______________________________________________ > Freeipa-devel mailing list > Freeipa-devel at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-devel From pvoborni at redhat.com Tue Sep 18 12:43:57 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Tue, 18 Sep 2012 14:43:57 +0200 Subject: [Freeipa-devel] [PATCH] 0073 Add trust verification code In-Reply-To: <20120918121539.GO28400@localhost.localdomain> References: <20120917154436.GA12295@redhat.com> <20120918104249.GN28400@localhost.localdomain> <20120918121539.GO28400@localhost.localdomain> Message-ID: <50586C8D.7050207@redhat.com> On 09/18/2012 02:15 PM, Sumit Bose wrote: > On Tue, Sep 18, 2012 at 12:42:49PM +0200, Sumit Bose wrote: >> On Mon, Sep 17, 2012 at 06:44:36PM +0300, Alexander Bokovoy wrote: >>> Hi, >>> >>> Following patch adds trust verification sequence to the case when we >>> establish trust with knowledge of AD administrative credentials. >>> >>> As we found out, in order to validate/verify trust, one has to have >>> administrative credentials for the trusted domain, since there are >>> few RPCs that should be performed against trusted domain's DC's LSA >>> and NetLogon pipes and these are protected by administrative credentials. >>> >>> Thus, when we know admin credentials for the remote domain, we can >>> perform the trust validation. >>> >>> https://fedorahosted.org/freeipa/ticket/2763 >>> >> >> Just a short feedback. The patch is working as expected, for a newly >> created trust Windows will send a TGS request to the IPA KDC without >> explicit validation on the windows side. Currently I have some issues >> in my test setup so that I can not give a full ACK atm. >> > > ok, ACK. > > Nevertheless it would be nice if Petr can check for any implications to > the web UI with respect to the status of the trust. It shouldn't break Web UI but Web UI won't use it. In add command Web UI uses only the command state (success/error). If the truststatus text would be a part of command summary text, it can be displayed in notification message (which fades after 3s) when comment 8 of https://fedorahosted.org/freeipa/ticket/2977#comment:8 is implemented. It would be nice if it can be saved to ldap and return in show/find commands? That way we can show it in search or details page. Or we can implement trust-status $TRUST --admin $ADMIN --$password $PASSWORD command to check the actual status anytime in a future. > > bye, > Sumit > >> bye, >> Sumit >> >>> >>> -- >>> / Alexander Bokovoy >> >> _______________________________________________ >> Freeipa-devel mailing list >> Freeipa-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/freeipa-devel > > _______________________________________________ > Freeipa-devel mailing list > Freeipa-devel at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-devel > -- Petr Vobornik From abokovoy at redhat.com Tue Sep 18 13:22:45 2012 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Tue, 18 Sep 2012 16:22:45 +0300 Subject: [Freeipa-devel] [PATCH] 0073 Add trust verification code In-Reply-To: <50586C8D.7050207@redhat.com> References: <20120917154436.GA12295@redhat.com> <20120918104249.GN28400@localhost.localdomain> <20120918121539.GO28400@localhost.localdomain> <50586C8D.7050207@redhat.com> Message-ID: <20120918132245.GC12295@redhat.com> On Tue, 18 Sep 2012, Petr Vobornik wrote: >On 09/18/2012 02:15 PM, Sumit Bose wrote: >>On Tue, Sep 18, 2012 at 12:42:49PM +0200, Sumit Bose wrote: >>>On Mon, Sep 17, 2012 at 06:44:36PM +0300, Alexander Bokovoy wrote: >>>>Hi, >>>> >>>>Following patch adds trust verification sequence to the case when we >>>>establish trust with knowledge of AD administrative credentials. >>>> >>>>As we found out, in order to validate/verify trust, one has to have >>>>administrative credentials for the trusted domain, since there are >>>>few RPCs that should be performed against trusted domain's DC's LSA >>>>and NetLogon pipes and these are protected by administrative credentials. >>>> >>>>Thus, when we know admin credentials for the remote domain, we can >>>>perform the trust validation. >>>> >>>>https://fedorahosted.org/freeipa/ticket/2763 >>>> >>> >>>Just a short feedback. The patch is working as expected, for a newly >>>created trust Windows will send a TGS request to the IPA KDC without >>>explicit validation on the windows side. Currently I have some issues >>>in my test setup so that I can not give a full ACK atm. >>> >> >>ok, ACK. >> >>Nevertheless it would be nice if Petr can check for any implications to >>the web UI with respect to the status of the trust. > >It shouldn't break Web UI but Web UI won't use it. In add command Web >UI uses only the command state (success/error). If the truststatus >text would be a part of command summary text, it can be displayed in >notification message (which fades after 3s) when comment 8 of >https://fedorahosted.org/freeipa/ticket/2977#comment:8 is >implemented. It is displayed as part of the output, truststatus property: # ipa trust-add --type=ad --admin Administrator at ad.local --password ad.local Active directory domain adminstrator's password: ------------------------------------------------- Added Active Directory trust for realm "ad.local" ------------------------------------------------- Realm name: ad.local Domain NetBIOS name: AD Domain Security Identifier: S-1-5-21-16904141-148189700-2149043814 Trust direction: Two-way trust Trust type: Active Directory domain Trust status: Established and verified Would be good if you could take it in use. >It would be nice if it can be saved to ldap and return in show/find >commands? That way we can show it in search or details page. Or we >can implement trust-status $TRUST --admin $ADMIN --$password >$PASSWORD command to check the actual status anytime in a future. We don't have an attribute to store the status. Neither it exists in Windows. I'll talk to Simo if we can have one attribute like that but the price of maintaining it up to date might be too much. On the other hand, we can always invalidate value in the attribute when ipasam cannot use shared trust account against trusted domain... Running validation/verification as a separate command is possible but it would be relatively resource-hungry and makes little use on its own. We may couple it together with future multiple suffixes support (tickets #2848, #2593) as fetching additional suffixes depends on validated trust relationship. -- / Alexander Bokovoy From pvoborni at redhat.com Tue Sep 18 15:22:26 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Tue, 18 Sep 2012 17:22:26 +0200 Subject: [Freeipa-devel] [PATCH] 0073 Add trust verification code In-Reply-To: <20120918132245.GC12295@redhat.com> References: <20120917154436.GA12295@redhat.com> <20120918104249.GN28400@localhost.localdomain> <20120918121539.GO28400@localhost.localdomain> <50586C8D.7050207@redhat.com> <20120918132245.GC12295@redhat.com> Message-ID: <505891B2.90408@redhat.com> On 09/18/2012 03:22 PM, Alexander Bokovoy wrote: > On Tue, 18 Sep 2012, Petr Vobornik wrote: >> On 09/18/2012 02:15 PM, Sumit Bose wrote: >>> On Tue, Sep 18, 2012 at 12:42:49PM +0200, Sumit Bose wrote: >>>> On Mon, Sep 17, 2012 at 06:44:36PM +0300, Alexander Bokovoy wrote: >>>>> Hi, >>>>> >>>>> Following patch adds trust verification sequence to the case when we >>>>> establish trust with knowledge of AD administrative credentials. >>>>> >>>>> As we found out, in order to validate/verify trust, one has to have >>>>> administrative credentials for the trusted domain, since there are >>>>> few RPCs that should be performed against trusted domain's DC's LSA >>>>> and NetLogon pipes and these are protected by administrative >>>>> credentials. >>>>> >>>>> Thus, when we know admin credentials for the remote domain, we can >>>>> perform the trust validation. >>>>> >>>>> https://fedorahosted.org/freeipa/ticket/2763 >>>>> >>>> >>>> Just a short feedback. The patch is working as expected, for a newly >>>> created trust Windows will send a TGS request to the IPA KDC without >>>> explicit validation on the windows side. Currently I have some issues >>>> in my test setup so that I can not give a full ACK atm. >>>> >>> >>> ok, ACK. >>> >>> Nevertheless it would be nice if Petr can check for any implications to >>> the web UI with respect to the status of the trust. >> >> It shouldn't break Web UI but Web UI won't use it. In add command Web >> UI uses only the command state (success/error). If the truststatus >> text would be a part of command summary text, it can be displayed in >> notification message (which fades after 3s) when comment 8 of >> https://fedorahosted.org/freeipa/ticket/2977#comment:8 is implemented. > It is displayed as part of the output, truststatus property: > # ipa trust-add --type=ad --admin Administrator at ad.local --password > ad.local > Active directory domain adminstrator's password: > ------------------------------------------------- > Added Active Directory trust for realm "ad.local" > ------------------------------------------------- > Realm name: ad.local > Domain NetBIOS name: AD > Domain Security Identifier: S-1-5-21-16904141-148189700-2149043814 > Trust direction: Two-way trust > Trust type: Active Directory domain > Trust status: Established and verified > > Would be good if you could take it in use. I created a patch which uses it. See attached screenshots. It may be useful but, as I wrote, the message is displayed only for 3s, so some users might not have time to read it whole - message is too long. >> It would be nice if it can be saved to ldap and return in show/find >> commands? That way we can show it in search or details page. Or we can >> implement trust-status $TRUST --admin $ADMIN --$password $PASSWORD >> command to check the actual status anytime in a future. > We don't have an attribute to store the status. Neither it exists in > Windows. > > I'll talk to Simo if we can have one attribute like that but the price > of maintaining it up to date might be too much. On the other hand, we > can always invalidate value in the attribute when ipasam cannot use > shared trust account against trusted domain... > > Running validation/verification as a separate command is possible but it > would be relatively resource-hungry and makes little use on its own. We > may couple it together with future multiple suffixes support (tickets > #2848, #2593) as fetching additional suffixes depends on validated trust > relationship. > -- Petr Vobornik -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pvoborni-0215-Show-trust-status-in-add-success-notification.patch Type: text/x-patch Size: 3564 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: trust-add-msg-established.png Type: image/png Size: 14543 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: trust-add-msg-waiting.png Type: image/png Size: 12863 bytes Desc: not available URL: From abokovoy at redhat.com Tue Sep 18 15:33:14 2012 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Tue, 18 Sep 2012 18:33:14 +0300 Subject: [Freeipa-devel] [PATCH] 0073 Add trust verification code In-Reply-To: <505891B2.90408@redhat.com> References: <20120917154436.GA12295@redhat.com> <20120918104249.GN28400@localhost.localdomain> <20120918121539.GO28400@localhost.localdomain> <50586C8D.7050207@redhat.com> <20120918132245.GC12295@redhat.com> <505891B2.90408@redhat.com> Message-ID: <20120918153314.GI12295@redhat.com> On Tue, 18 Sep 2012, Petr Vobornik wrote: >On 09/18/2012 03:22 PM, Alexander Bokovoy wrote: >>On Tue, 18 Sep 2012, Petr Vobornik wrote: >>>On 09/18/2012 02:15 PM, Sumit Bose wrote: >>>>On Tue, Sep 18, 2012 at 12:42:49PM +0200, Sumit Bose wrote: >>>>>On Mon, Sep 17, 2012 at 06:44:36PM +0300, Alexander Bokovoy wrote: >>>>>>Hi, >>>>>> >>>>>>Following patch adds trust verification sequence to the case when we >>>>>>establish trust with knowledge of AD administrative credentials. >>>>>> >>>>>>As we found out, in order to validate/verify trust, one has to have >>>>>>administrative credentials for the trusted domain, since there are >>>>>>few RPCs that should be performed against trusted domain's DC's LSA >>>>>>and NetLogon pipes and these are protected by administrative >>>>>>credentials. >>>>>> >>>>>>Thus, when we know admin credentials for the remote domain, we can >>>>>>perform the trust validation. >>>>>> >>>>>>https://fedorahosted.org/freeipa/ticket/2763 >>>>>> >>>>> >>>>>Just a short feedback. The patch is working as expected, for a newly >>>>>created trust Windows will send a TGS request to the IPA KDC without >>>>>explicit validation on the windows side. Currently I have some issues >>>>>in my test setup so that I can not give a full ACK atm. >>>>> >>>> >>>>ok, ACK. >>>> >>>>Nevertheless it would be nice if Petr can check for any implications to >>>>the web UI with respect to the status of the trust. >>> >>>It shouldn't break Web UI but Web UI won't use it. In add command Web >>>UI uses only the command state (success/error). If the truststatus >>>text would be a part of command summary text, it can be displayed in >>>notification message (which fades after 3s) when comment 8 of >>>https://fedorahosted.org/freeipa/ticket/2977#comment:8 is implemented. >>It is displayed as part of the output, truststatus property: >># ipa trust-add --type=ad --admin Administrator at ad.local --password >>ad.local >>Active directory domain adminstrator's password: >>------------------------------------------------- >>Added Active Directory trust for realm "ad.local" >>------------------------------------------------- >> Realm name: ad.local >> Domain NetBIOS name: AD >> Domain Security Identifier: S-1-5-21-16904141-148189700-2149043814 >> Trust direction: Two-way trust >> Trust type: Active Directory domain >> Trust status: Established and verified >> >>Would be good if you could take it in use. > >I created a patch which uses it. See attached screenshots. It may be >useful but, as I wrote, the message is displayed only for 3s, so some >users might not have time to read it whole - message is too long. Well, as we don't have other means to show this information right now, that's good too. Maybe notification message timer could be possible to tune per instance? Then we could have, say, 5 seconds timeout here and keep 3 seconds as default one... -- / Alexander Bokovoy From pvoborni at redhat.com Tue Sep 18 15:54:08 2012 From: pvoborni at redhat.com (Petr Vobornik) Date: Tue, 18 Sep 2012 17:54:08 +0200 Subject: [Freeipa-devel] [PATCH] 0073 Add trust verification code In-Reply-To: <20120918153314.GI12295@redhat.com> References: <20120917154436.GA12295@redhat.com> <20120918104249.GN28400@localhost.localdomain> <20120918121539.GO28400@localhost.localdomain> <50586C8D.7050207@redhat.com> <20120918132245.GC12295@redhat.com> <505891B2.90408@redhat.com> <20120918153314.GI12295@redhat.com> Message-ID: <50589920.7020402@redhat.com> On 09/18/2012 05:33 PM, Alexander Bokovoy wrote: > On Tue, 18 Sep 2012, Petr Vobornik wrote: >> On 09/18/2012 03:22 PM, Alexander Bokovoy wrote: >>> On Tue, 18 Sep 2012, Petr Vobornik wrote: >>>> On 09/18/2012 02:15 PM, Sumit Bose wrote: >>>>> On Tue, Sep 18, 2012 at 12:42:49PM +0200, Sumit Bose wrote: >>>>>> On Mon, Sep 17, 2012 at 06:44:36PM +0300, Alexander Bokovoy wrote: >>>>>>> Hi, >>>>>>> >>>>>>> Following patch adds trust verification sequence to the case when we >>>>>>> establish trust with knowledge of AD administrative credentials. >>>>>>> >>>>>>> As we found out, in order to validate/verify trust, one has to have >>>>>>> administrative credentials for the trusted domain, since there are >>>>>>> few RPCs that should be performed against trusted domain's DC's LSA >>>>>>> and NetLogon pipes and these are protected by administrative >>>>>>> credentials. >>>>>>> >>>>>>> Thus, when we know admin credentials for the remote domain, we can >>>>>>> perform the trust validation. >>>>>>> >>>>>>> https://fedorahosted.org/freeipa/ticket/2763 >>>>>>> >>>>>> >>>>>> Just a short feedback. The patch is working as expected, for a newly >>>>>> created trust Windows will send a TGS request to the IPA KDC without >>>>>> explicit validation on the windows side. Currently I have some issues >>>>>> in my test setup so that I can not give a full ACK atm. >>>>>> >>>>> >>>>> ok, ACK. >>>>> >>>>> Nevertheless it would be nice if Petr can check for any >>>>> implications to >>>>> the web UI with respect to the status of the trust. >>>> >>>> It shouldn't break Web UI but Web UI won't use it. In add command Web >>>> UI uses only the command state (success/error). If the truststatus >>>> text would be a part of command summary text, it can be displayed in >>>> notification message (which fades after 3s) when comment 8 of >>>> https://fedorahosted.org/freeipa/ticket/2977#comment:8 is implemented. >>> It is displayed as part of the output, truststatus property: >>> # ipa trust-add --type=ad --admin Administrator at ad.local --password >>> ad.local >>> Active directory domain adminstrator's password: >>> ------------------------------------------------- >>> Added Active Directory trust for realm "ad.local" >>> ------------------------------------------------- >>> Realm name: ad.local >>> Domain NetBIOS name: AD >>> Domain Security Identifier: S-1-5-21-16904141-148189700-2149043814 >>> Trust direction: Two-way trust >>> Trust type: Active Directory domain >>> Trust status: Established and verified >>> >>> Would be good if you could take it in use. >> >> I created a patch which uses it. See attached screenshots. It may be >> useful but, as I wrote, the message is displayed only for 3s, so some >> users might not have time to read it whole - message is too long. > Well, as we don't have other means to show this information right now, > that's good too. Maybe notification message timer could be possible to > tune per instance? Then we could have, say, 5 seconds timeout here and > keep 3 seconds as default one... > I tuned it. Updated patch attached. -- Petr Vobornik -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pvoborni-0215-1-Show-trust-status-in-add-success-notification.patch Type: text/x-patch Size: 4687 bytes Desc: not available URL: From rcritten at redhat.com Tue Sep 18 16:08:34 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 18 Sep 2012 12:08:34 -0400 Subject: [Freeipa-devel] [PATCH] 0081 Only stop the main DS instance when upgrading it In-Reply-To: <50534F3B.6030603@redhat.com> References: <505328D1.2020806@redhat.com> <1347628337.19861.31.camel@willson.li.ssimo.org> <50533B69.2030905@redhat.com> <505344F0.3020901@redhat.com> <50534F3B.6030603@redhat.com> Message-ID: <50589C82.2050108@redhat.com> Martin Kosek wrote: > On 09/14/2012 04:53 PM, Petr Viktorin wrote: >> On 09/14/2012 04:12 PM, Petr Viktorin wrote: >>> On 09/14/2012 03:12 PM, Simo Sorce wrote: >>>> On Fri, 2012-09-14 at 14:53 +0200, Petr Viktorin wrote: >>>>> This fixes a 2.2?3.0 upgrade bug found while testing the Dogtag 10 work. >>>>> See commit or ticket for details. >>>>> >>>>> https://fedorahosted.org/freeipa/ticket/3083 >>>>> >>>>> >>>>> >>>>> I also suspect that waiting for ports is not a good way to check if the >>>>> CMS is fully initialized, but I don't know of a better way. If you know >>>>> one, please speak up. >>>>> >> >> Such a better way is coming with https://fedorahosted.org/pki/ticket/314. I >> opened https://fedorahosted.org/freeipa/ticket/3084 so we won't forget to take >> advantage of it. >> >>>> Won;t we get back to square zero when the work to merge DS and CS >>>> instances into one will be completed ? >>>> >>>> Simo. >>>> >>> >>> When they're merged we'll probably need to bring down the CA while >>> upgrading the server. Or just stop all the IPA services to be on the >>> safe side, and of course bring them back up afterwards. >>> Meanwhile, this patch shouldn't hurt things. > > The patch worked fine for me. It will be useful at least to the point when we > use a common DS instance, as Simo suggested. Useful for now, pushed to master and ipa-3-0. rob From abokovoy at redhat.com Tue Sep 18 16:08:57 2012 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Tue, 18 Sep 2012 19:08:57 +0300 Subject: [Freeipa-devel] [PATCH] 0073 Add trust verification code In-Reply-To: <50589920.7020402@redhat.com> References: <20120917154436.GA12295@redhat.com> <20120918104249.GN28400@localhost.localdomain> <20120918121539.GO28400@localhost.localdomain> <50586C8D.7050207@redhat.com> <20120918132245.GC12295@redhat.com> <505891B2.90408@redhat.com> <20120918153314.GI12295@redhat.com> <50589920.7020402@redhat.com> Message-ID: <20120918160857.GL12295@redhat.com> On Tue, 18 Sep 2012, Petr Vobornik wrote: >On 09/18/2012 05:33 PM, Alexander Bokovoy wrote: >>On Tue, 18 Sep 2012, Petr Vobornik wrote: >>>On 09/18/2012 03:22 PM, Alexander Bokovoy wrote: >>>>On Tue, 18 Sep 2012, Petr Vobornik wrote: >>>>>On 09/18/2012 02:15 PM, Sumit Bose wrote: >>>>>>On Tue, Sep 18, 2012 at 12:42:49PM +0200, Sumit Bose wrote: >>>>>>>On Mon, Sep 17, 2012 at 06:44:36PM +0300, Alexander Bokovoy wrote: >>>>>>>>Hi, >>>>>>>> >>>>>>>>Following patch adds trust verification sequence to the case when we >>>>>>>>establish trust with knowledge of AD administrative credentials. >>>>>>>> >>>>>>>>As we found out, in order to validate/verify trust, one has to have >>>>>>>>administrative credentials for the trusted domain, since there are >>>>>>>>few RPCs that should be performed against trusted domain's DC's LSA >>>>>>>>and NetLogon pipes and these are protected by administrative >>>>>>>>credentials. >>>>>>>> >>>>>>>>Thus, when we know admin credentials for the remote domain, we can >>>>>>>>perform the trust validation. >>>>>>>> >>>>>>>>https://fedorahosted.org/freeipa/ticket/2763 >>>>>>>> >>>>>>> >>>>>>>Just a short feedback. The patch is working as expected, for a newly >>>>>>>created trust Windows will send a TGS request to the IPA KDC without >>>>>>>explicit validation on the windows side. Currently I have some issues >>>>>>>in my test setup so that I can not give a full ACK atm. >>>>>>> >>>>>> >>>>>>ok, ACK. >>>>>> >>>>>>Nevertheless it would be nice if Petr can check for any >>>>>>implications to >>>>>>the web UI with respect to the status of the trust. >>>>> >>>>>It shouldn't break Web UI but Web UI won't use it. In add command Web >>>>>UI uses only the command state (success/error). If the truststatus >>>>>text would be a part of command summary text, it can be displayed in >>>>>notification message (which fades after 3s) when comment 8 of >>>>>https://fedorahosted.org/freeipa/ticket/2977#comment:8 is implemented. >>>>It is displayed as part of the output, truststatus property: >>>># ipa trust-add --type=ad --admin Administrator at ad.local --password >>>>ad.local >>>>Active directory domain adminstrator's password: >>>>------------------------------------------------- >>>>Added Active Directory trust for realm "ad.local" >>>>------------------------------------------------- >>>> Realm name: ad.local >>>> Domain NetBIOS name: AD >>>> Domain Security Identifier: S-1-5-21-16904141-148189700-2149043814 >>>> Trust direction: Two-way trust >>>> Trust type: Active Directory domain >>>> Trust status: Established and verified >>>> >>>>Would be good if you could take it in use. >>> >>>I created a patch which uses it. See attached screenshots. It may be >>>useful but, as I wrote, the message is displayed only for 3s, so some >>>users might not have time to read it whole - message is too long. >>Well, as we don't have other means to show this information right now, >>that's good too. Maybe notification message timer could be possible to >>tune per instance? Then we could have, say, 5 seconds timeout here and >>keep 3 seconds as default one... >> > >I tuned it. Updated patch attached. ACK. Worked fine for me. -- / Alexander Bokovoy From edewata at redhat.com Tue Sep 18 16:47:31 2012 From: edewata at redhat.com (Endi Sukma Dewata) Date: Tue, 18 Sep 2012 11:47:31 -0500 Subject: [Freeipa-devel] [PATCH] 212 Fix integer validation when boundary value is empty string In-Reply-To: <50585CAD.5030905@redhat.com> References: <504F542A.7060204@redhat.com> <5057E046.3030303@redhat.com> <50585CAD.5030905@redhat.com> Message-ID: <5058A5A3.7090105@redhat.com> On 9/18/2012 6:36 AM, Petr Vobornik wrote: > Updated patch attached. ACK. >> 1. Instead of IPA.not_defined() it might be better to call it >> IPA.defined() to avoid double negations like this: >> >> if (!IPA.not_defined(metadata.minvalue, true) ... > > Function renamed, logic negated. Unit tests for IPA.defined added. > >> 2. The check_empty_str probably could made optional and the value would >> be true by default. It will make the code cleaner. > > I wanted the function to be more general - used somewhere else in a > future. In general, I consider empty string as a defined value so for me > 'false' is a good default value of the argument. I agree that in this > case it would look cleaner. OK. It's also possible to use different methods, e.g. IPA.is_null() and IPA.is_empty(). -- Endi S. Dewata From rcritten at redhat.com Tue Sep 18 18:42:32 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Tue, 18 Sep 2012 14:42:32 -0400 Subject: [Freeipa-devel] [PATCH] 0073 Add trust verification code In-Reply-To: <20120918160857.GL12295@redhat.com> References: <20120917154436.GA12295@redhat.com> <20120918104249.GN28400@localhost.localdomain> <20120918121539.GO28400@localhost.localdomain> <50586C8D.7050207@redhat.com> <20120918132245.GC12295@redhat.com> <505891B2.90408@redhat.com> <20120918153314.GI12295@redhat.com> <50589920.7020402@redhat.com> <20120918160857.GL12295@redhat.com> Message-ID: <5058C098.2020800@redhat.com> Alexander Bokovoy wrote: > On Tue, 18 Sep 2012, Petr Vobornik wrote: >> On 09/18/2012 05:33 PM, Alexander Bokovoy wrote: >>> On Tue, 18 Sep 2012, Petr Vobornik wrote: >>>> On 09/18/2012 03:22 PM, Alexander Bokovoy wrote: >>>>> On Tue, 18 Sep 2012, Petr Vobornik wrote: >>>>>> On 09/18/2012 02:15 PM, Sumit Bose wrote: >>>>>>> On Tue, Sep 18, 2012 at 12:42:49PM +0200, Sumit Bose wrote: >>>>>>>> On Mon, Sep 17, 2012 at 06:44:36PM +0300, Alexander Bokovoy wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> Following patch adds trust verification sequence to the case >>>>>>>>> when we >>>>>>>>> establish trust with knowledge of AD administrative credentials. >>>>>>>>> >>>>>>>>> As we found out, in order to validate/verify trust, one has to >>>>>>>>> have >>>>>>>>> administrative credentials for the trusted domain, since there are >>>>>>>>> few RPCs that should be performed against trusted domain's DC's >>>>>>>>> LSA >>>>>>>>> and NetLogon pipes and these are protected by administrative >>>>>>>>> credentials. >>>>>>>>> >>>>>>>>> Thus, when we know admin credentials for the remote domain, we can >>>>>>>>> perform the trust validation. >>>>>>>>> >>>>>>>>> https://fedorahosted.org/freeipa/ticket/2763 >>>>>>>>> >>>>>>>> >>>>>>>> Just a short feedback. The patch is working as expected, for a >>>>>>>> newly >>>>>>>> created trust Windows will send a TGS request to the IPA KDC >>>>>>>> without >>>>>>>> explicit validation on the windows side. Currently I have some >>>>>>>> issues >>>>>>>> in my test setup so that I can not give a full ACK atm. >>>>>>>> >>>>>>> >>>>>>> ok, ACK. >>>>>>> >>>>>>> Nevertheless it would be nice if Petr can check for any >>>>>>> implications to >>>>>>> the web UI with respect to the status of the trust. >>>>>> >>>>>> It shouldn't break Web UI but Web UI won't use it. In add command Web >>>>>> UI uses only the command state (success/error). If the truststatus >>>>>> text would be a part of command summary text, it can be displayed in >>>>>> notification message (which fades after 3s) when comment 8 of >>>>>> https://fedorahosted.org/freeipa/ticket/2977#comment:8 is >>>>>> implemented. >>>>> It is displayed as part of the output, truststatus property: >>>>> # ipa trust-add --type=ad --admin Administrator at ad.local --password >>>>> ad.local >>>>> Active directory domain adminstrator's password: >>>>> ------------------------------------------------- >>>>> Added Active Directory trust for realm "ad.local" >>>>> ------------------------------------------------- >>>>> Realm name: ad.local >>>>> Domain NetBIOS name: AD >>>>> Domain Security Identifier: S-1-5-21-16904141-148189700-2149043814 >>>>> Trust direction: Two-way trust >>>>> Trust type: Active Directory domain >>>>> Trust status: Established and verified >>>>> >>>>> Would be good if you could take it in use. >>>> >>>> I created a patch which uses it. See attached screenshots. It may be >>>> useful but, as I wrote, the message is displayed only for 3s, so some >>>> users might not have time to read it whole - message is too long. >>> Well, as we don't have other means to show this information right now, >>> that's good too. Maybe notification message timer could be possible to >>> tune per instance? Then we could have, say, 5 seconds timeout here and >>> keep 3 seconds as default one... >>> >> >> I tuned it. Updated patch attached. > ACK. Worked fine for me. > Pushed 073 and 215.1 to ipa-3-0 and master rob From mkosek at redhat.com Wed Sep 19 11:58:52 2012 From: mkosek at redhat.com (Martin Kosek) Date: Wed, 19 Sep 2012 13:58:52 +0200 Subject: [Freeipa-devel] Integration of dogtag10 Message-ID: <5059B37C.3060603@redhat.com> Hello Ade, I am continuing a testing of integration of dogtag10 + ipa 3.0 on Fedora 18. After defining the missing apache-commons-codec.jar link + permissive SELinux, clean IPA installation + cert operations works fine. I just hit an issue when installing an F18+dogtag10 replica for a F17+dogtag9 master. Is this scenario still unsupported or is it a bug? (I am moving the whole dogtag10 development discussion outside of the previous long thread since your and pviktori's patches have been accepted and pushed). ipa-replica-install output: # ipa-replica-install -p Secret123 -w Secret123 --setup-ca ~/replica-info-vm-021.idm.lab.bos.redhat.com.gpg Run connection check to master Check connection from replica to remote master 'vm-086.idm.lab.bos.redhat.com': ... Connection from master to replica is OK. Connection check OK Configuring ntpd [1/4]: stopping ntpd [2/4]: writing configuration [3/4]: configuring ntpd to start on boot [4/4]: starting ntpd done configuring ntpd. Configuring directory server for the CA: Estimated time 30 seconds [1/3]: creating directory server user [2/3]: creating directory server instance [3/3]: restarting directory server done configuring pkids. Configuring certificate server: Estimated time 3 minutes 30 seconds [1/15]: creating certificate server user [2/15]: configuring certificate server instance Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up. Unexpected error - see /var/log/ipareplica-install.log for details: IOError: [Errno 2] No such file or directory: '/var/lib/pki/pki-tomcat/alias/ca_backup_keys.p12' ipa+pki logs attached. Thanks, Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: dogtag10-replica.tgz Type: application/x-compressed-tar Size: 28779 bytes Desc: not available URL: From alee at redhat.com Wed Sep 19 14:16:17 2012 From: alee at redhat.com (Ade Lee) Date: Wed, 19 Sep 2012 10:16:17 -0400 Subject: [Freeipa-devel] Integration of dogtag10 In-Reply-To: <5059B37C.3060603@redhat.com> References: <5059B37C.3060603@redhat.com> Message-ID: <1348064178.2624.30.camel@aleeredhat.laptop> Its a bug. Basically, the d10 instance is trying to get an installation token from the security domain, using a new restful interface. This, on a dogtag 9 instance, results in a 404. We need to change the d10 code to fall back to the old interface in case the new one does not work. https://fedorahosted.org/pki/ticket/334 has been created on dogtag trac, Ade On Wed, 2012-09-19 at 13:58 +0200, Martin Kosek wrote: > Hello Ade, > > I am continuing a testing of integration of dogtag10 + ipa 3.0 on Fedora 18. > After defining the missing apache-commons-codec.jar link + permissive SELinux, > clean IPA installation + cert operations works fine. I just hit an issue when > installing an F18+dogtag10 replica for a F17+dogtag9 master. > > Is this scenario still unsupported or is it a bug? > > (I am moving the whole dogtag10 development discussion outside of the previous > long thread since your and pviktori's patches have been accepted and pushed). > > ipa-replica-install output: > > # ipa-replica-install -p Secret123 -w Secret123 --setup-ca > ~/replica-info-vm-021.idm.lab.bos.redhat.com.gpg > Run connection check to master > Check connection from replica to remote master 'vm-086.idm.lab.bos.redhat.com': > ... > > Connection from master to replica is OK. > > Connection check OK > Configuring ntpd > [1/4]: stopping ntpd > [2/4]: writing configuration > [3/4]: configuring ntpd to start on boot > [4/4]: starting ntpd > done configuring ntpd. > Configuring directory server for the CA: Estimated time 30 seconds > [1/3]: creating directory server user > [2/3]: creating directory server instance > [3/3]: restarting directory server > done configuring pkids. > Configuring certificate server: Estimated time 3 minutes 30 seconds > [1/15]: creating certificate server user > [2/15]: configuring certificate server instance > > Your system may be partly configured. > Run /usr/sbin/ipa-server-install --uninstall to clean up. > > Unexpected error - see /var/log/ipareplica-install.log for details: > IOError: [Errno 2] No such file or directory: > '/var/lib/pki/pki-tomcat/alias/ca_backup_keys.p12' > > > ipa+pki logs attached. > > Thanks, > Martin From alee at redhat.com Wed Sep 19 14:18:47 2012 From: alee at redhat.com (Ade Lee) Date: Wed, 19 Sep 2012 10:18:47 -0400 Subject: [Freeipa-devel] Integration of dogtag10 In-Reply-To: <5059B37C.3060603@redhat.com> References: <5059B37C.3060603@redhat.com> Message-ID: <1348064328.2624.31.camel@aleeredhat.laptop> Its a bug. Basically, the d10 instance is trying to get an installation token from the security domain, using a new restful interface. This, on a dogtag 9 instance, results in a 404. We need to change the d10 code to fall back to the old interface in case the new one does not work. https://fedorahosted.org/pki/ticket/334 has been created on dogtag trac, Ade On Wed, 2012-09-19 at 13:58 +0200, Martin Kosek wrote: > Hello Ade, > > I am continuing a testing of integration of dogtag10 + ipa 3.0 on Fedora 18. > After defining the missing apache-commons-codec.jar link + permissive SELinux, > clean IPA installation + cert operations works fine. I just hit an issue when > installing an F18+dogtag10 replica for a F17+dogtag9 master. > > Is this scenario still unsupported or is it a bug? > > (I am moving the whole dogtag10 development discussion outside of the previous > long thread since your and pviktori's patches have been accepted and pushed). > > ipa-replica-install output: > > # ipa-replica-install -p Secret123 -w Secret123 --setup-ca > ~/replica-info-vm-021.idm.lab.bos.redhat.com.gpg > Run connection check to master > Check connection from replica to remote master 'vm-086.idm.lab.bos.redhat.com': > ... > > Connection from master to replica is OK. > > Connection check OK > Configuring ntpd > [1/4]: stopping ntpd > [2/4]: writing configuration > [3/4]: configuring ntpd to start on boot > [4/4]: starting ntpd > done configuring ntpd. > Configuring directory server for the CA: Estimated time 30 seconds > [1/3]: creating directory server user > [2/3]: creating directory server instance > [3/3]: restarting directory server > done configuring pkids. > Configuring certificate server: Estimated time 3 minutes 30 seconds > [1/15]: creating certificate server user > [2/15]: configuring certificate server instance > > Your system may be partly configured. > Run /usr/sbin/ipa-server-install --uninstall to clean up. > > Unexpected error - see /var/log/ipareplica-install.log for details: > IOError: [Errno 2] No such file or directory: > '/var/lib/pki/pki-tomcat/alias/ca_backup_keys.p12' > > > ipa+pki logs attached. > > Thanks, > Martin From alee at redhat.com Wed Sep 19 14:19:35 2012 From: alee at redhat.com (Ade Lee) Date: Wed, 19 Sep 2012 10:19:35 -0400 Subject: [Freeipa-devel] Integration of dogtag10 In-Reply-To: <5059B37C.3060603@redhat.com> References: <5059B37C.3060603@redhat.com> Message-ID: <1348064375.2575.0.camel@aleeredhat.laptop> Its a bug. Basically, the d10 instance is trying to get an installation token from the security domain, using a new restful interface. This, on a dogtag 9 instance, results in a 404. We need to change the d10 code to fall back to the old interface in case the new one does not work. Ade On Wed, 2012-09-19 at 13:58 +0200, Martin Kosek wrote: > Hello Ade, > > I am continuing a testing of integration of dogtag10 + ipa 3.0 on Fedora 18. > After defining the missing apache-commons-codec.jar link + permissive SELinux, > clean IPA installation + cert operations works fine. I just hit an issue when > installing an F18+dogtag10 replica for a F17+dogtag9 master. > > Is this scenario still unsupported or is it a bug? > > (I am moving the whole dogtag10 development discussion outside of the previous > long thread since your and pviktori's patches have been accepted and pushed). > > ipa-replica-install output: > > # ipa-replica-install -p Secret123 -w Secret123 --setup-ca > ~/replica-info-vm-021.idm.lab.bos.redhat.com.gpg > Run connection check to master > Check connection from replica to remote master 'vm-086.idm.lab.bos.redhat.com': > ... > > Connection from master to replica is OK. > > Connection check OK > Configuring ntpd > [1/4]: stopping ntpd > [2/4]: writing configuration > [3/4]: configuring ntpd to start on boot > [4/4]: starting ntpd > done configuring ntpd. > Configuring directory server for the CA: Estimated time 30 seconds > [1/3]: creating directory server user > [2/3]: creating directory server instance > [3/3]: restarting directory server > done configuring pkids. > Configuring certificate server: Estimated time 3 minutes 30 seconds > [1/15]: creating certificate server user > [2/15]: configuring certificate server instance > > Your system may be partly configured. > Run /usr/sbin/ipa-server-install --uninstall to clean up. > > Unexpected error - see /var/log/ipareplica-install.log for details: > IOError: [Errno 2] No such file or directory: > '/var/lib/pki/pki-tomcat/alias/ca_backup_keys.p12' > > > ipa+pki logs attached. > > Thanks, > Martin From rcritten at redhat.com Wed Sep 19 14:56:22 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 19 Sep 2012 10:56:22 -0400 Subject: [Freeipa-devel] [PATCH] 0077 Check direct/reverse hostname/address resolution in ipa-replica-install In-Reply-To: <50585832.8060005@redhat.com> References: <5046009E.2020306@redhat.com> <50463DE6.9070300@redhat.com> <5046FF0E.1000504@redhat.com> <504FA7A3.5080201@redhat.com> <5050B1ED.70307@redhat.com> <50524387.4020800@redhat.com> <5052D2CC.2000405@redhat.com> <50570D68.9040501@redhat.com> <505767AA.5080607@redhat.com> <50585832.8060005@redhat.com> Message-ID: <5059DD16.9010202@redhat.com> Petr Viktorin wrote: > On 09/17/2012 08:10 PM, Rob Crittenden wrote: >> Petr Viktorin wrote: >>> On 09/14/2012 08:46 AM, Martin Kosek wrote: >>>> On 09/13/2012 10:35 PM, Rob Crittenden wrote: >>>>> Petr Viktorin wrote: >>>>>> On 09/11/2012 11:05 PM, Rob Crittenden wrote: >>>>>>> Petr Viktorin wrote: >>>>>>>> On 09/04/2012 07:44 PM, Rob Crittenden wrote: >>>>>>>>> Petr Viktorin wrote: >>>>>>>>>> >>>>>>>>>> https://fedorahosted.org/freeipa/ticket/2845 >>>>>>>>> >>>>>>>>> Shouldn't this also call verify_fqdn() on the local hostname and >>>>>>>>> not >>>>>>>>> just the master? I think this would eventually fail in the >>>>>>>>> conncheck >>>>>>>>> but >>>>>>>>> what if that was skipped? >>>>>>>>> >>>>>>>>> rob >>>>>>>> >>>>>>>> A few lines above there is a call to get_host_name, which will call >>>>>>>> verify_fqdn. >>>>>>>> >>>>>>> >>>>>>> I double-checked this, it fails in conncheck. Here are my steps: >>>>>>> >>>>>>> # ipa-server-install --setup-dns >>>>>>> # ipa-replica-prepare replica.example.com --ip-address=192.168.100.2 >>>>>>> # ipa host-del replica.example.com >>>>>>> >>>>>>> On replica, set DNS to IPA master, with hostname in /etc/hosts. >>>>>>> >>>>>>> # ipa-replica-install ... >>>>>>> >>>>>>> The verify_fqdn() passes because the resolver uses /etc/hosts. >>>>>>> >>>>>>> The conncheck fails: >>>>>>> >>>>>>> Execute check on remote master >>>>>>> Check connection from master to remote replica >>>>>>> 'replica.example.com': >>>>>>> >>>>>>> Remote master check failed with following error message(s): >>>>>>> Could not chdir to home directory /home/admin: No such file or >>>>>>> directory >>>>>>> Port check failed! Unable to resolve host name 'replica.example.com' >>>>>>> >>>>>>> Connection check failed! >>>>>>> Please fix your network settings according to error messages above. >>>>>>> If the check results are not valid it can be skipped with >>>>>>> --skip-conncheck parameter. >>>>>>> >>>>>>> The DNS test happens much further after this, and I get why, I just >>>>>>> don't see how useful it is unless the --skip-conncheck is used. >>>>>> >>>>>> For the record, it's because we need to check if the host has DNS >>>>>> installed. We need a LDAP connection to check this. >>>>>> >>>>>>> ipa-replica-install ~rcrit/replica-info-replica.example.com.gpg >>>>>>> --skip-conncheck >>>>>>> Directory Manager (existing master) password: >>>>>>> >>>>>>> ipa : ERROR Could not resolve hostname >>>>>>> replica.example.com >>>>>>> using DNS. Clients may not function properly. Please check your DNS >>>>>>> setup. (Note that this check queries IPA DNS directly and ignores >>>>>>> /etc/hosts.) >>>>>>> Continue? [no]: >>>>>>> >>>>>>> So I guess, what are the intentions here? It is certainly better >>>>>>> than >>>>>>> before. >>>>>>> >>>>>>> rob >>>>>> >>>>>> If the replica is in the master's /etc/hosts, but not in DNS, the >>>>>> conncheck will succeed. This check explicitly queries IPA records >>>>>> only >>>>>> and ignores /etc/hosts so it'll notice this case and warn. >>>>>> >>>>> >>>>> Ok, like I said, this is better than we have. Just one nit then you >>>>> get an ack: >>>>> >>>>> + # If remote host has DNS, check forward/reverse resolution >>>>> + try: >>>>> + entry = conn.find_entries(u'cn=dns', >>>>> base_dn=DN(api.env.basedn)) >>>>> + except errors.NotFound: >>>>> >>>>> u'cn=dns' should be str(constants.container_dns). >>>>> >>>>> rob >>>> >>>> This is a search filter, Petr could use the one I already have in >>>> "dns.py::get_dns_masters()" function: >>>> '(&(objectClass=ipaConfigObject)(cn=DNS))' >>>> >>>> For performance sake, I would also not search in the entire tree, but >>>> limit the >>>> search only to: >>>> >>>> DN(('cn', 'masters'), ('cn', 'ipa'), ('cn', 'etc'), api.env.basedn) >>>> >>>> Martin >>>> >>> >>> Attaching updated patch with Martin's suggestions. >> >> I think what Martin had in mind was: >> >> if api.Object.dnsrecord.get_dns_masters(): >> ... >> > > I didn't want to do this because api.Object.* use our global ldap2 > Backend, which is hardwired to query localhost. > I see now that I can hack around this, and we already do this in > ipa-replica-install. > I've extracted the hack and reused it to get the DNS masters. > > I can't say I'm crazy about the method name you've chosen... rob From mkosek at redhat.com Wed Sep 19 15:06:18 2012 From: mkosek at redhat.com (Martin Kosek) Date: Wed, 19 Sep 2012 17:06:18 +0200 Subject: [Freeipa-devel] [PATCH] 302 Stricter IP network validator in dnszone-add command In-Reply-To: <50577B87.3080204@redhat.com> References: <5047077E.1040000@redhat.com> <50471BBD.1070506@redhat.com> <504727E5.5040101@redhat.com> <50472B1C.2070207@redhat.com> <50472DF6.8000207@redhat.com> <5047314B.30201@redhat.com> <5051D666.1060604@redhat.com> <50577B87.3080204@redhat.com> Message-ID: <5059DF6A.3090805@redhat.com> On 09/17/2012 09:35 PM, Rob Crittenden wrote: > Martin Kosek wrote: >> On 09/05/2012 01:02 PM, Jan Cholasta wrote: >>> Dne 5.9.2012 12:48, Martin Kosek napsal(a): >>>> On 09/05/2012 12:36 PM, Jan Cholasta wrote: >>>>> Dne 5.9.2012 12:22, Petr Spacek napsal(a): >>>>>> On 09/05/2012 11:30 AM, Jan Cholasta wrote: >>>>>>> Dne 5.9.2012 10:04, Martin Kosek napsal(a): >>>>>>>> We allowed IP addresses without network specification which lead >>>>>>>> to unexpected results when the zone was being created. We should rather >>>>>>>> strictly require the prefix/netmask specifying the IP network that >>>>>>>> the reverse zone should be created for. This is already done in >>>>>>>> Web UI. >>>>>>>> >>>>>>>> A unit test exercising this new validation was added. >>>>>>>> >>>>>>>> https://fedorahosted.org/freeipa/ticket/2461 >>>>>>>> >>>>>>> >>>>>>> I don't like this much. I would suggest using CheckedIPAddress and not >>>>>>> forcing >>>>>>> the user to enter the prefix length instead. >>>>>>> >>>>>>> CheckedIPAddress uses a sensible default prefix length if one is not >>>>>>> specified >>>>>>> (class-based for IPv4, /64 for IPv6) as opposed to IPNetwork (/32 for >>>>>>> IPv4, >>>>>>> /128 for IPv6 - this causes the erroneous reverse zones to be created as >>>>>>> described in the ticket). >>>>>>> >>>>>> Hello, >>>>>> >>>>>> I don't like automatic netmask guessing. I have met class-based guessing >>>>>> in Windows (XP?) and I was forced to overwrite default mask all the time >>>>>> ... >>>>> >>>>> If there was no guessing, you would have to write the netmask anyway, so I >>>>> don't see any harm in guessing here. >>>>> >>>>>> >>>>>> IMHO there is no "sensible default prefix" in real world. I sitting on >>>>>> network with /23 prefix right now. Also, I have never seen 10.x network >>>>>> with /8 prefix. >>>>>> >>>>> >>>>> While this might be true for IPv4 in some cases, /64 is perfectly sensible >>>>> for >>>>> IPv6. Also, I have never seen 192.168.x.x network with non-/24 prefix. >>>>> >>>>> Honza >>>>> >>>> >>>> While this may be true for 192.168.x.x, it does not apply for 10.x.x.x >>>> networks >>>> as Petr already pointed out. I don't think that there will be many people >>>> expecting that a reverse zone of 10.0.0.0/24 would be created. >>> >>> And they would be correct, because the default prefix length for a class A >>> network is /8, not /24. >>> >>>> >>>> And since FreeIPA is mainly deployed to internal networks, I assume this will >>>> be the case of most users. >>>> >>>> Martin >>>> >>> >>> OK, but what about IPv6? Correct me if I'm wrong, but the prefix length is >>> going to be /64 99% of the time for IPv6. >>> >>> The installer uses /24 for IPv4 addresses and /64 for IPv6 addresses, maybe >>> this should be used as a default here as well. >>> >>> Honza >>> >> >> In the end, I choose a more liberal approach and instead of defining a more >> stricter validator for IPv4 only I rather used approach already implemented in >> the installers, i.e. default length of network prefix is 24 for IPv4 and 64 for >> IPv6. >> >> Updated patch attached. >> >> Martin > > Works for me. I wonder if this is a candidate for some more unit tests... > > rob > One more test should not hurt. Updated patch attached. Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-302-3-use-default-reverse-zone-consistently.patch Type: text/x-patch Size: 11803 bytes Desc: not available URL: From rcritten at redhat.com Wed Sep 19 15:30:46 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 19 Sep 2012 11:30:46 -0400 Subject: [Freeipa-devel] [PATCH] 302 Stricter IP network validator in dnszone-add command In-Reply-To: <5059DF6A.3090805@redhat.com> References: <5047077E.1040000@redhat.com> <50471BBD.1070506@redhat.com> <504727E5.5040101@redhat.com> <50472B1C.2070207@redhat.com> <50472DF6.8000207@redhat.com> <5047314B.30201@redhat.com> <5051D666.1060604@redhat.com> <50577B87.3080204@redhat.com> <5059DF6A.3090805@redhat.com> Message-ID: <5059E526.80804@redhat.com> Martin Kosek wrote: > On 09/17/2012 09:35 PM, Rob Crittenden wrote: >> Martin Kosek wrote: >>> On 09/05/2012 01:02 PM, Jan Cholasta wrote: >>>> Dne 5.9.2012 12:48, Martin Kosek napsal(a): >>>>> On 09/05/2012 12:36 PM, Jan Cholasta wrote: >>>>>> Dne 5.9.2012 12:22, Petr Spacek napsal(a): >>>>>>> On 09/05/2012 11:30 AM, Jan Cholasta wrote: >>>>>>>> Dne 5.9.2012 10:04, Martin Kosek napsal(a): >>>>>>>>> We allowed IP addresses without network specification which lead >>>>>>>>> to unexpected results when the zone was being created. We should rather >>>>>>>>> strictly require the prefix/netmask specifying the IP network that >>>>>>>>> the reverse zone should be created for. This is already done in >>>>>>>>> Web UI. >>>>>>>>> >>>>>>>>> A unit test exercising this new validation was added. >>>>>>>>> >>>>>>>>> https://fedorahosted.org/freeipa/ticket/2461 >>>>>>>>> >>>>>>>> >>>>>>>> I don't like this much. I would suggest using CheckedIPAddress and not >>>>>>>> forcing >>>>>>>> the user to enter the prefix length instead. >>>>>>>> >>>>>>>> CheckedIPAddress uses a sensible default prefix length if one is not >>>>>>>> specified >>>>>>>> (class-based for IPv4, /64 for IPv6) as opposed to IPNetwork (/32 for >>>>>>>> IPv4, >>>>>>>> /128 for IPv6 - this causes the erroneous reverse zones to be created as >>>>>>>> described in the ticket). >>>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I don't like automatic netmask guessing. I have met class-based guessing >>>>>>> in Windows (XP?) and I was forced to overwrite default mask all the time >>>>>>> ... >>>>>> >>>>>> If there was no guessing, you would have to write the netmask anyway, so I >>>>>> don't see any harm in guessing here. >>>>>> >>>>>>> >>>>>>> IMHO there is no "sensible default prefix" in real world. I sitting on >>>>>>> network with /23 prefix right now. Also, I have never seen 10.x network >>>>>>> with /8 prefix. >>>>>>> >>>>>> >>>>>> While this might be true for IPv4 in some cases, /64 is perfectly sensible >>>>>> for >>>>>> IPv6. Also, I have never seen 192.168.x.x network with non-/24 prefix. >>>>>> >>>>>> Honza >>>>>> >>>>> >>>>> While this may be true for 192.168.x.x, it does not apply for 10.x.x.x >>>>> networks >>>>> as Petr already pointed out. I don't think that there will be many people >>>>> expecting that a reverse zone of 10.0.0.0/24 would be created. >>>> >>>> And they would be correct, because the default prefix length for a class A >>>> network is /8, not /24. >>>> >>>>> >>>>> And since FreeIPA is mainly deployed to internal networks, I assume this will >>>>> be the case of most users. >>>>> >>>>> Martin >>>>> >>>> >>>> OK, but what about IPv6? Correct me if I'm wrong, but the prefix length is >>>> going to be /64 99% of the time for IPv6. >>>> >>>> The installer uses /24 for IPv4 addresses and /64 for IPv6 addresses, maybe >>>> this should be used as a default here as well. >>>> >>>> Honza >>>> >>> >>> In the end, I choose a more liberal approach and instead of defining a more >>> stricter validator for IPv4 only I rather used approach already implemented in >>> the installers, i.e. default length of network prefix is 24 for IPv4 and 64 for >>> IPv6. >>> >>> Updated patch attached. >>> >>> Martin >> >> Works for me. I wonder if this is a candidate for some more unit tests... >> >> rob >> > > One more test should not hurt. Updated patch attached. > > Martin > ACK From mkosek at redhat.com Wed Sep 19 15:32:49 2012 From: mkosek at redhat.com (Martin Kosek) Date: Wed, 19 Sep 2012 17:32:49 +0200 Subject: [Freeipa-devel] [PATCH] 302 Stricter IP network validator in dnszone-add command In-Reply-To: <5059E526.80804@redhat.com> References: <5047077E.1040000@redhat.com> <50471BBD.1070506@redhat.com> <504727E5.5040101@redhat.com> <50472B1C.2070207@redhat.com> <50472DF6.8000207@redhat.com> <5047314B.30201@redhat.com> <5051D666.1060604@redhat.com> <50577B87.3080204@redhat.com> <5059DF6A.3090805@redhat.com> <5059E526.80804@redhat.com> Message-ID: <5059E5A1.6020907@redhat.com> On 09/19/2012 05:30 PM, Rob Crittenden wrote: > Martin Kosek wrote: >> On 09/17/2012 09:35 PM, Rob Crittenden wrote: >>> Martin Kosek wrote: >>>> On 09/05/2012 01:02 PM, Jan Cholasta wrote: >>>>> Dne 5.9.2012 12:48, Martin Kosek napsal(a): >>>>>> On 09/05/2012 12:36 PM, Jan Cholasta wrote: >>>>>>> Dne 5.9.2012 12:22, Petr Spacek napsal(a): >>>>>>>> On 09/05/2012 11:30 AM, Jan Cholasta wrote: >>>>>>>>> Dne 5.9.2012 10:04, Martin Kosek napsal(a): >>>>>>>>>> We allowed IP addresses without network specification which lead >>>>>>>>>> to unexpected results when the zone was being created. We should rather >>>>>>>>>> strictly require the prefix/netmask specifying the IP network that >>>>>>>>>> the reverse zone should be created for. This is already done in >>>>>>>>>> Web UI. >>>>>>>>>> >>>>>>>>>> A unit test exercising this new validation was added. >>>>>>>>>> >>>>>>>>>> https://fedorahosted.org/freeipa/ticket/2461 >>>>>>>>>> >>>>>>>>> >>>>>>>>> I don't like this much. I would suggest using CheckedIPAddress and not >>>>>>>>> forcing >>>>>>>>> the user to enter the prefix length instead. >>>>>>>>> >>>>>>>>> CheckedIPAddress uses a sensible default prefix length if one is not >>>>>>>>> specified >>>>>>>>> (class-based for IPv4, /64 for IPv6) as opposed to IPNetwork (/32 for >>>>>>>>> IPv4, >>>>>>>>> /128 for IPv6 - this causes the erroneous reverse zones to be created as >>>>>>>>> described in the ticket). >>>>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> I don't like automatic netmask guessing. I have met class-based guessing >>>>>>>> in Windows (XP?) and I was forced to overwrite default mask all the time >>>>>>>> ... >>>>>>> >>>>>>> If there was no guessing, you would have to write the netmask anyway, so I >>>>>>> don't see any harm in guessing here. >>>>>>> >>>>>>>> >>>>>>>> IMHO there is no "sensible default prefix" in real world. I sitting on >>>>>>>> network with /23 prefix right now. Also, I have never seen 10.x network >>>>>>>> with /8 prefix. >>>>>>>> >>>>>>> >>>>>>> While this might be true for IPv4 in some cases, /64 is perfectly sensible >>>>>>> for >>>>>>> IPv6. Also, I have never seen 192.168.x.x network with non-/24 prefix. >>>>>>> >>>>>>> Honza >>>>>>> >>>>>> >>>>>> While this may be true for 192.168.x.x, it does not apply for 10.x.x.x >>>>>> networks >>>>>> as Petr already pointed out. I don't think that there will be many people >>>>>> expecting that a reverse zone of 10.0.0.0/24 would be created. >>>>> >>>>> And they would be correct, because the default prefix length for a class A >>>>> network is /8, not /24. >>>>> >>>>>> >>>>>> And since FreeIPA is mainly deployed to internal networks, I assume this >>>>>> will >>>>>> be the case of most users. >>>>>> >>>>>> Martin >>>>>> >>>>> >>>>> OK, but what about IPv6? Correct me if I'm wrong, but the prefix length is >>>>> going to be /64 99% of the time for IPv6. >>>>> >>>>> The installer uses /24 for IPv4 addresses and /64 for IPv6 addresses, maybe >>>>> this should be used as a default here as well. >>>>> >>>>> Honza >>>>> >>>> >>>> In the end, I choose a more liberal approach and instead of defining a more >>>> stricter validator for IPv4 only I rather used approach already implemented in >>>> the installers, i.e. default length of network prefix is 24 for IPv4 and 64 >>>> for >>>> IPv6. >>>> >>>> Updated patch attached. >>>> >>>> Martin >>> >>> Works for me. I wonder if this is a candidate for some more unit tests... >>> >>> rob >>> >> >> One more test should not hurt. Updated patch attached. >> >> Martin >> > > ACK Pushed to master, ipa-3-0. Martin From pviktori at redhat.com Wed Sep 19 15:44:59 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Wed, 19 Sep 2012 17:44:59 +0200 Subject: [Freeipa-devel] [PATCH] 0077 Check direct/reverse hostname/address resolution in ipa-replica-install In-Reply-To: <5059DD16.9010202@redhat.com> References: <5046009E.2020306@redhat.com> <50463DE6.9070300@redhat.com> <5046FF0E.1000504@redhat.com> <504FA7A3.5080201@redhat.com> <5050B1ED.70307@redhat.com> <50524387.4020800@redhat.com> <5052D2CC.2000405@redhat.com> <50570D68.9040501@redhat.com> <505767AA.5080607@redhat.com> <50585832.8060005@redhat.com> <5059DD16.9010202@redhat.com> Message-ID: <5059E87B.2070500@redhat.com> On 09/19/2012 04:56 PM, Rob Crittenden wrote: > Petr Viktorin wrote: >> On 09/17/2012 08:10 PM, Rob Crittenden wrote: >>> Petr Viktorin wrote: >>>> On 09/14/2012 08:46 AM, Martin Kosek wrote: >>>>> On 09/13/2012 10:35 PM, Rob Crittenden wrote: >>>>>> Petr Viktorin wrote: >>>>>>> On 09/11/2012 11:05 PM, Rob Crittenden wrote: >>>>>>>> Petr Viktorin wrote: >>>>>>>>> On 09/04/2012 07:44 PM, Rob Crittenden wrote: >>>>>>>>>> Petr Viktorin wrote: >>>>>>>>>>> >>>>>>>>>>> https://fedorahosted.org/freeipa/ticket/2845 >>>>>>>>>> >>>>>>>>>> Shouldn't this also call verify_fqdn() on the local hostname and >>>>>>>>>> not >>>>>>>>>> just the master? I think this would eventually fail in the >>>>>>>>>> conncheck >>>>>>>>>> but >>>>>>>>>> what if that was skipped? >>>>>>>>>> >>>>>>>>>> rob >>>>>>>>> >>>>>>>>> A few lines above there is a call to get_host_name, which will >>>>>>>>> call >>>>>>>>> verify_fqdn. >>>>>>>>> >>>>>>>> >>>>>>>> I double-checked this, it fails in conncheck. Here are my steps: >>>>>>>> >>>>>>>> # ipa-server-install --setup-dns >>>>>>>> # ipa-replica-prepare replica.example.com >>>>>>>> --ip-address=192.168.100.2 >>>>>>>> # ipa host-del replica.example.com >>>>>>>> >>>>>>>> On replica, set DNS to IPA master, with hostname in /etc/hosts. >>>>>>>> >>>>>>>> # ipa-replica-install ... >>>>>>>> >>>>>>>> The verify_fqdn() passes because the resolver uses /etc/hosts. >>>>>>>> >>>>>>>> The conncheck fails: >>>>>>>> >>>>>>>> Execute check on remote master >>>>>>>> Check connection from master to remote replica >>>>>>>> 'replica.example.com': >>>>>>>> >>>>>>>> Remote master check failed with following error message(s): >>>>>>>> Could not chdir to home directory /home/admin: No such file or >>>>>>>> directory >>>>>>>> Port check failed! Unable to resolve host name >>>>>>>> 'replica.example.com' >>>>>>>> >>>>>>>> Connection check failed! >>>>>>>> Please fix your network settings according to error messages above. >>>>>>>> If the check results are not valid it can be skipped with >>>>>>>> --skip-conncheck parameter. >>>>>>>> >>>>>>>> The DNS test happens much further after this, and I get why, I just >>>>>>>> don't see how useful it is unless the --skip-conncheck is used. >>>>>>> >>>>>>> For the record, it's because we need to check if the host has DNS >>>>>>> installed. We need a LDAP connection to check this. >>>>>>> >>>>>>>> ipa-replica-install ~rcrit/replica-info-replica.example.com.gpg >>>>>>>> --skip-conncheck >>>>>>>> Directory Manager (existing master) password: >>>>>>>> >>>>>>>> ipa : ERROR Could not resolve hostname >>>>>>>> replica.example.com >>>>>>>> using DNS. Clients may not function properly. Please check your DNS >>>>>>>> setup. (Note that this check queries IPA DNS directly and ignores >>>>>>>> /etc/hosts.) >>>>>>>> Continue? [no]: >>>>>>>> >>>>>>>> So I guess, what are the intentions here? It is certainly better >>>>>>>> than >>>>>>>> before. >>>>>>>> >>>>>>>> rob >>>>>>> >>>>>>> If the replica is in the master's /etc/hosts, but not in DNS, the >>>>>>> conncheck will succeed. This check explicitly queries IPA records >>>>>>> only >>>>>>> and ignores /etc/hosts so it'll notice this case and warn. >>>>>>> >>>>>> >>>>>> Ok, like I said, this is better than we have. Just one nit then you >>>>>> get an ack: >>>>>> >>>>>> + # If remote host has DNS, check forward/reverse resolution >>>>>> + try: >>>>>> + entry = conn.find_entries(u'cn=dns', >>>>>> base_dn=DN(api.env.basedn)) >>>>>> + except errors.NotFound: >>>>>> >>>>>> u'cn=dns' should be str(constants.container_dns). >>>>>> >>>>>> rob >>>>> >>>>> This is a search filter, Petr could use the one I already have in >>>>> "dns.py::get_dns_masters()" function: >>>>> '(&(objectClass=ipaConfigObject)(cn=DNS))' >>>>> >>>>> For performance sake, I would also not search in the entire tree, but >>>>> limit the >>>>> search only to: >>>>> >>>>> DN(('cn', 'masters'), ('cn', 'ipa'), ('cn', 'etc'), api.env.basedn) >>>>> >>>>> Martin >>>>> >>>> >>>> Attaching updated patch with Martin's suggestions. >>> >>> I think what Martin had in mind was: >>> >>> if api.Object.dnsrecord.get_dns_masters(): >>> ... >>> >> >> I didn't want to do this because api.Object.* use our global ldap2 >> Backend, which is hardwired to query localhost. >> I see now that I can hack around this, and we already do this in >> ipa-replica-install. >> I've extracted the hack and reused it to get the DNS masters. >> >> > > I can't say I'm crazy about the method name you've chosen... > > rob I intended the name as a warning to not use it unless necessary. Changed to temporary_ldap2_connection. -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0077-04-Check-direct-reverse-hostname-address-resolution-in-.patch Type: text/x-patch Size: 9985 bytes Desc: not available URL: From abokovoy at redhat.com Wed Sep 19 16:19:10 2012 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Wed, 19 Sep 2012 19:19:10 +0300 Subject: [Freeipa-devel] [PATCH] 0074 validate SID for trusted domain when adding/modifying ID range Message-ID: <20120919161910.GB24531@redhat.com> Hi, This patch adds validation of SID for trusted domain when adding or modifying ID range for the domain. We only allow creating ranges for trusted domains when the trust is already established -- the default range is created automatically right after the trust is added. https://fedorahosted.org/freeipa/ticket/3087 -- / Alexander Bokovoy -------------- next part -------------- >From c8859d449b65be67841c96c81f7f64f8c27b06b1 Mon Sep 17 00:00:00 2001 From: Alexander Bokovoy Date: Wed, 19 Sep 2012 19:09:22 +0300 Subject: [PATCH] validate SID for trusted domain when adding/modifying ID range https://fedorahosted.org/freeipa/ticket/3087 --- ipalib/plugins/idrange.py | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/ipalib/plugins/idrange.py b/ipalib/plugins/idrange.py index efa906428aa58c670bc4af63b10c88123dda5b65..4750c1d6716bd69045d53f32ae1836f44e70b03b 100644 --- a/ipalib/plugins/idrange.py +++ b/ipalib/plugins/idrange.py @@ -26,6 +26,12 @@ from ipapython import ipautil from ipalib import util from ipapython.dn import DN +if api.env.in_server and api.env.context in ['lite', 'server']: + try: + import ipaserver.dcerpc + _dcerpc_bindings_installed = True + except Exception, e: + _dcerpc_bindings_installed = False __doc__ = _(""" ID ranges @@ -137,6 +143,21 @@ user. RIDs are unique in a domain, 32bit values and are used for users and groups. """) +def validate_trusted_domain_sid(self, sid): + if not _dcerpc_bindings_installed: + raise errors.NotFound(name=_('ID Range setup'), + reason=_('''Cannot perform SID validation without Samba 4 support installed. + Make sure you have installed server-trust-ad sub-package of IPA on the server''')) + domain_validator = ipaserver.dcerpc.DomainValidator(self.api) + if not domain_validator.is_configured(): + raise errors.NotFound(name=_('ID Range setup'), + reason=_('''Cross-realm trusts are not configured.. + Make sure you have run ipa-adtrust-install on the IPA server first''')) + if not domain_validator.is_trusted_sid_valid(sid): + raise errors.ValidationError(name=_('ID Range setup'), + error=_('SID is not recognized as a valid SID from a trusted domain')) + + class idrange(LDAPObject): """ Range object. @@ -287,6 +308,9 @@ class idrange_add(LDAPCreate): error=_('Options dom_sid and rid_base must ' \ 'be used together')) + # Validate SID as the one of trusted domains + validate_trusted_domain_sid(self, options['ipanttrusteddomainsid']) + # Finally, add trusted AD domain range object class entry_attrs['objectclass'].append('ipatrustedaddomainrange') else: if (('ipasecondarybaserid' in options) != ('ipabaserid' in options)): @@ -366,6 +390,10 @@ class idrange_mod(LDAPUpdate): except errors.NotFound: self.obj.handle_not_found(*keys) + if 'ipanttrusteddomainsid' in options: + # Validate SID as the one of trusted domains + validate_trusted_domain_sid(self, options['ipanttrusteddomainsid']) + old_base_id = int(old_attrs.get('ipabaseid', [0])[0]) old_range_size = int(old_attrs.get('ipaidrangesize', [0])[0]) new_base_id = entry_attrs.get('ipabaseid') -- 1.7.12 From rcritten at redhat.com Wed Sep 19 18:46:49 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 19 Sep 2012 14:46:49 -0400 Subject: [Freeipa-devel] [PATCH] 0077 Check direct/reverse hostname/address resolution in ipa-replica-install In-Reply-To: <5059E87B.2070500@redhat.com> References: <5046009E.2020306@redhat.com> <50463DE6.9070300@redhat.com> <5046FF0E.1000504@redhat.com> <504FA7A3.5080201@redhat.com> <5050B1ED.70307@redhat.com> <50524387.4020800@redhat.com> <5052D2CC.2000405@redhat.com> <50570D68.9040501@redhat.com> <505767AA.5080607@redhat.com> <50585832.8060005@redhat.com> <5059DD16.9010202@redhat.com> <5059E87B.2070500@redhat.com> Message-ID: <505A1319.8010803@redhat.com> Petr Viktorin wrote: > On 09/19/2012 04:56 PM, Rob Crittenden wrote: >> Petr Viktorin wrote: >>> On 09/17/2012 08:10 PM, Rob Crittenden wrote: >>>> Petr Viktorin wrote: >>>>> On 09/14/2012 08:46 AM, Martin Kosek wrote: >>>>>> On 09/13/2012 10:35 PM, Rob Crittenden wrote: >>>>>>> Petr Viktorin wrote: >>>>>>>> On 09/11/2012 11:05 PM, Rob Crittenden wrote: >>>>>>>>> Petr Viktorin wrote: >>>>>>>>>> On 09/04/2012 07:44 PM, Rob Crittenden wrote: >>>>>>>>>>> Petr Viktorin wrote: >>>>>>>>>>>> >>>>>>>>>>>> https://fedorahosted.org/freeipa/ticket/2845 >>>>>>>>>>> >>>>>>>>>>> Shouldn't this also call verify_fqdn() on the local hostname and >>>>>>>>>>> not >>>>>>>>>>> just the master? I think this would eventually fail in the >>>>>>>>>>> conncheck >>>>>>>>>>> but >>>>>>>>>>> what if that was skipped? >>>>>>>>>>> >>>>>>>>>>> rob >>>>>>>>>> >>>>>>>>>> A few lines above there is a call to get_host_name, which will >>>>>>>>>> call >>>>>>>>>> verify_fqdn. >>>>>>>>>> >>>>>>>>> >>>>>>>>> I double-checked this, it fails in conncheck. Here are my steps: >>>>>>>>> >>>>>>>>> # ipa-server-install --setup-dns >>>>>>>>> # ipa-replica-prepare replica.example.com >>>>>>>>> --ip-address=192.168.100.2 >>>>>>>>> # ipa host-del replica.example.com >>>>>>>>> >>>>>>>>> On replica, set DNS to IPA master, with hostname in /etc/hosts. >>>>>>>>> >>>>>>>>> # ipa-replica-install ... >>>>>>>>> >>>>>>>>> The verify_fqdn() passes because the resolver uses /etc/hosts. >>>>>>>>> >>>>>>>>> The conncheck fails: >>>>>>>>> >>>>>>>>> Execute check on remote master >>>>>>>>> Check connection from master to remote replica >>>>>>>>> 'replica.example.com': >>>>>>>>> >>>>>>>>> Remote master check failed with following error message(s): >>>>>>>>> Could not chdir to home directory /home/admin: No such file or >>>>>>>>> directory >>>>>>>>> Port check failed! Unable to resolve host name >>>>>>>>> 'replica.example.com' >>>>>>>>> >>>>>>>>> Connection check failed! >>>>>>>>> Please fix your network settings according to error messages >>>>>>>>> above. >>>>>>>>> If the check results are not valid it can be skipped with >>>>>>>>> --skip-conncheck parameter. >>>>>>>>> >>>>>>>>> The DNS test happens much further after this, and I get why, I >>>>>>>>> just >>>>>>>>> don't see how useful it is unless the --skip-conncheck is used. >>>>>>>> >>>>>>>> For the record, it's because we need to check if the host has DNS >>>>>>>> installed. We need a LDAP connection to check this. >>>>>>>> >>>>>>>>> ipa-replica-install ~rcrit/replica-info-replica.example.com.gpg >>>>>>>>> --skip-conncheck >>>>>>>>> Directory Manager (existing master) password: >>>>>>>>> >>>>>>>>> ipa : ERROR Could not resolve hostname >>>>>>>>> replica.example.com >>>>>>>>> using DNS. Clients may not function properly. Please check your >>>>>>>>> DNS >>>>>>>>> setup. (Note that this check queries IPA DNS directly and ignores >>>>>>>>> /etc/hosts.) >>>>>>>>> Continue? [no]: >>>>>>>>> >>>>>>>>> So I guess, what are the intentions here? It is certainly better >>>>>>>>> than >>>>>>>>> before. >>>>>>>>> >>>>>>>>> rob >>>>>>>> >>>>>>>> If the replica is in the master's /etc/hosts, but not in DNS, the >>>>>>>> conncheck will succeed. This check explicitly queries IPA records >>>>>>>> only >>>>>>>> and ignores /etc/hosts so it'll notice this case and warn. >>>>>>>> >>>>>>> >>>>>>> Ok, like I said, this is better than we have. Just one nit then you >>>>>>> get an ack: >>>>>>> >>>>>>> + # If remote host has DNS, check forward/reverse resolution >>>>>>> + try: >>>>>>> + entry = conn.find_entries(u'cn=dns', >>>>>>> base_dn=DN(api.env.basedn)) >>>>>>> + except errors.NotFound: >>>>>>> >>>>>>> u'cn=dns' should be str(constants.container_dns). >>>>>>> >>>>>>> rob >>>>>> >>>>>> This is a search filter, Petr could use the one I already have in >>>>>> "dns.py::get_dns_masters()" function: >>>>>> '(&(objectClass=ipaConfigObject)(cn=DNS))' >>>>>> >>>>>> For performance sake, I would also not search in the entire tree, but >>>>>> limit the >>>>>> search only to: >>>>>> >>>>>> DN(('cn', 'masters'), ('cn', 'ipa'), ('cn', 'etc'), api.env.basedn) >>>>>> >>>>>> Martin >>>>>> >>>>> >>>>> Attaching updated patch with Martin's suggestions. >>>> >>>> I think what Martin had in mind was: >>>> >>>> if api.Object.dnsrecord.get_dns_masters(): >>>> ... >>>> >>> >>> I didn't want to do this because api.Object.* use our global ldap2 >>> Backend, which is hardwired to query localhost. >>> I see now that I can hack around this, and we already do this in >>> ipa-replica-install. >>> I've extracted the hack and reused it to get the DNS masters. >>> >>> >> >> I can't say I'm crazy about the method name you've chosen... >> >> rob > > I intended the name as a warning to not use it unless necessary. > > Changed to temporary_ldap2_connection. > I found a dangling reference to replman. I removed this and installation seemed to work ok. --- install/tools/ipa-replica-install 2012-09-19 14:01:16.169053047 -0400 +++ /usr/sbin/ipa-replica-install 2012-09-19 14:43:06.684917906 -0400 @@ -564,8 +564,6 @@ finally: if conn and conn.isconnected(): conn.disconnect() - if replman and replman.conn: - replman.conn.unbind_s() # Configure ntpd if options.conf_ntp: From alee at redhat.com Thu Sep 20 03:58:53 2012 From: alee at redhat.com (Ade Lee) Date: Wed, 19 Sep 2012 23:58:53 -0400 Subject: [Freeipa-devel] [PATCH] Changes to use a single database for dogtag and IPA Message-ID: <1348113533.2575.14.camel@aleeredhat.laptop> Changes to use a single database for dogtag and IPA New servers that are installed with dogtag 10 instances will use a single database instance for dogtag and IPA, albeit with different suffixes. Dogtag will communicate with the instance through a database user with permissions to modify the dogtag suffix only. This user will authenticate using client auth using the subsystem cert for the instance. This patch includes changes to allow the creation of masters and clones with single ds instances. I have tested being able to create a master and a clone using f17 and dogtag 10. Note that you will need to use the latest builds on the dogtag repo to get some changes that were checked in today. We'll kick off another official f18 dogtag build in a day or so. This is a pretty big change - so I expect many issues to come up as things get tested. But as this will take awhile to get resolved, its better to get this out for review as fast as possible. Happy reviewing. Ade -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Changes-to-use-a-single-database-for-dogtag-and-IPA.patch Type: text/x-patch Size: 32646 bytes Desc: not available URL: From jcholast at redhat.com Thu Sep 20 08:22:38 2012 From: jcholast at redhat.com (Jan Cholasta) Date: Thu, 20 Sep 2012 10:22:38 +0200 Subject: [Freeipa-devel] [PATCH] SSHPublicKey.fingerprint_dns_sha1 should return unicode value Message-ID: <505AD24E.5020407@redhat.com> Hi, this one-liner fixes updating DNS SSHFP records in host-mod. Honza -- Jan Cholasta -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-jcholast-86-SSHPublicKey.fingerprint_dns_sha1-should-return-unic.patch Type: text/x-patch Size: 667 bytes Desc: not available URL: From mkosek at redhat.com Thu Sep 20 08:25:07 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 20 Sep 2012 10:25:07 +0200 Subject: [Freeipa-devel] [PATCH] 0074 validate SID for trusted domain when adding/modifying ID range In-Reply-To: <20120919161910.GB24531@redhat.com> References: <20120919161910.GB24531@redhat.com> Message-ID: <505AD2E3.7090004@redhat.com> On 09/19/2012 06:19 PM, Alexander Bokovoy wrote: > Hi, > > This patch adds validation of SID for trusted domain when adding or > modifying ID range for the domain. We only allow creating ranges for > trusted domains when the trust is already established -- the default > range is created automatically right after the trust is added. > > https://fedorahosted.org/freeipa/ticket/3087 > Basic functionality looks OK, but I saw few issues with exception formatting: diff --git a/ipalib/plugins/idrange.py b/ipalib/plugins/idrange.py index efa906428aa58c670bc4af63b10c88123dda5b65..4750c1d6716bd69045d53f32ae1836f44e70b03b 100644 --- a/ipalib/plugins/idrange.py +++ b/ipalib/plugins/idrange.py @@ -26,6 +26,12 @@ from ipapython import ipautil from ipalib import util from ipapython.dn import DN +if api.env.in_server and api.env.context in ['lite', 'server']: + try: + import ipaserver.dcerpc + _dcerpc_bindings_installed = True + except Exception, e: + _dcerpc_bindings_installed = False Variable "e" is not used, so it can be removed. __doc__ = _(""" ID ranges @@ -137,6 +143,21 @@ user. RIDs are unique in a domain, 32bit values and are used for users and groups. """) +def validate_trusted_domain_sid(self, sid): "self" is not needed in the list of attributes, this is not a class method. + if not _dcerpc_bindings_installed: + raise errors.NotFound(name=_('ID Range setup'), + reason=_('''Cannot perform SID validation without Samba 4 support installed. + Make sure you have installed server-trust-ad sub-package of IPA on the server''')) Improperly formatted exception: 1) NotFound error does not use "name" param, maybe you wanted to use ValidationError? 2) The text will be improperly formatted - since you used '''''', the indentation will be in text: ipa: ERROR: Cannot perform SID validation without Samba 4 support installed. Make sure you have installed server-trust-ad sub-package of IPA on the server Also, I know this was discussed before, but using gettext in a name attribute of ValidationError will cause improperly formatted exception: # ipa idrange-add foo --base-id=1000 --range-size=100 --dom-sid=foo ipa: ERROR: invalid Gettext('ID Range setup', domain='ipa', localedir=None): Options dom_sid and rid_base must be used together The problem is, that "name" param is printer as %r, thus you would need to coerce it to unicode to make it better. + domain_validator = ipaserver.dcerpc.DomainValidator(self.api) + if not domain_validator.is_configured(): + raise errors.NotFound(name=_('ID Range setup'), + reason=_('''Cross-realm trusts are not configured.. + Make sure you have run ipa-adtrust-install on the IPA server first''')) Same issues: # ipa idrange-add foo --base-id=1000 --range-size=100 --dom-sid=foo --rid-base=1000 ipa: ERROR: Cross-realm trusts are not configured.. Make sure you have run ipa-adtrust-install on the IPA server first + if not domain_validator.is_trusted_sid_valid(sid): + raise errors.ValidationError(name=_('ID Range setup'), + error=_('SID is not recognized as a valid SID from a trusted domain')) + + Same issues: ipa: ERROR: invalid Gettext('ID Range setup', domain='ipa', localedir=None): SID is not recognized as a valid SID from a trusted domain Martin From mkosek at redhat.com Thu Sep 20 08:28:58 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 20 Sep 2012 10:28:58 +0200 Subject: [Freeipa-devel] [PATCH] 311 Fix idrange plugin help Message-ID: <505AD3CA.2060907@redhat.com> range plugin was renamed to idrange. Update plugin help to reflect this change. -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-311-fix-idrange-plugin-help.patch Type: text/x-patch Size: 1866 bytes Desc: not available URL: From pspacek at redhat.com Thu Sep 20 08:33:20 2012 From: pspacek at redhat.com (Petr Spacek) Date: Thu, 20 Sep 2012 10:33:20 +0200 Subject: [Freeipa-devel] [PATCH] 311 Fix idrange plugin help In-Reply-To: <505AD3CA.2060907@redhat.com> References: <505AD3CA.2060907@redhat.com> Message-ID: <505AD4D0.7050102@redhat.com> On 09/20/2012 10:28 AM, Martin Kosek wrote: > range plugin was renamed to idrange. Update plugin help to reflect > this change. ACK :-) Petr^2 Spacek From pviktori at redhat.com Thu Sep 20 08:32:59 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Thu, 20 Sep 2012 10:32:59 +0200 Subject: [Freeipa-devel] [PATCH] 0077 Check direct/reverse hostname/address resolution in ipa-replica-install In-Reply-To: <505A1319.8010803@redhat.com> References: <5046009E.2020306@redhat.com> <50463DE6.9070300@redhat.com> <5046FF0E.1000504@redhat.com> <504FA7A3.5080201@redhat.com> <5050B1ED.70307@redhat.com> <50524387.4020800@redhat.com> <5052D2CC.2000405@redhat.com> <50570D68.9040501@redhat.com> <505767AA.5080607@redhat.com> <50585832.8060005@redhat.com> <5059DD16.9010202@redhat.com> <5059E87B.2070500@redhat.com> <505A1319.8010803@redhat.com> Message-ID: <505AD4BB.9040509@redhat.com> On 09/19/2012 08:46 PM, Rob Crittenden wrote: > Petr Viktorin wrote: >> On 09/19/2012 04:56 PM, Rob Crittenden wrote: >>> Petr Viktorin wrote: >>>> On 09/17/2012 08:10 PM, Rob Crittenden wrote: >>>>> Petr Viktorin wrote: >>>>>> On 09/14/2012 08:46 AM, Martin Kosek wrote: >>>>>>> On 09/13/2012 10:35 PM, Rob Crittenden wrote: >>>>>>>> Petr Viktorin wrote: >>>>>>>>> On 09/11/2012 11:05 PM, Rob Crittenden wrote: >>>>>>>>>> Petr Viktorin wrote: >>>>>>>>>>> On 09/04/2012 07:44 PM, Rob Crittenden wrote: >>>>>>>>>>>> Petr Viktorin wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> https://fedorahosted.org/freeipa/ticket/2845 >>>>>>>>>>>> >>>>>>>>>>>> Shouldn't this also call verify_fqdn() on the local hostname >>>>>>>>>>>> and >>>>>>>>>>>> not >>>>>>>>>>>> just the master? I think this would eventually fail in the >>>>>>>>>>>> conncheck >>>>>>>>>>>> but >>>>>>>>>>>> what if that was skipped? >>>>>>>>>>>> >>>>>>>>>>>> rob >>>>>>>>>>> >>>>>>>>>>> A few lines above there is a call to get_host_name, which will >>>>>>>>>>> call >>>>>>>>>>> verify_fqdn. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I double-checked this, it fails in conncheck. Here are my steps: >>>>>>>>>> >>>>>>>>>> # ipa-server-install --setup-dns >>>>>>>>>> # ipa-replica-prepare replica.example.com >>>>>>>>>> --ip-address=192.168.100.2 >>>>>>>>>> # ipa host-del replica.example.com >>>>>>>>>> >>>>>>>>>> On replica, set DNS to IPA master, with hostname in /etc/hosts. >>>>>>>>>> >>>>>>>>>> # ipa-replica-install ... >>>>>>>>>> >>>>>>>>>> The verify_fqdn() passes because the resolver uses /etc/hosts. >>>>>>>>>> >>>>>>>>>> The conncheck fails: >>>>>>>>>> >>>>>>>>>> Execute check on remote master >>>>>>>>>> Check connection from master to remote replica >>>>>>>>>> 'replica.example.com': >>>>>>>>>> >>>>>>>>>> Remote master check failed with following error message(s): >>>>>>>>>> Could not chdir to home directory /home/admin: No such file or >>>>>>>>>> directory >>>>>>>>>> Port check failed! Unable to resolve host name >>>>>>>>>> 'replica.example.com' >>>>>>>>>> >>>>>>>>>> Connection check failed! >>>>>>>>>> Please fix your network settings according to error messages >>>>>>>>>> above. >>>>>>>>>> If the check results are not valid it can be skipped with >>>>>>>>>> --skip-conncheck parameter. >>>>>>>>>> >>>>>>>>>> The DNS test happens much further after this, and I get why, I >>>>>>>>>> just >>>>>>>>>> don't see how useful it is unless the --skip-conncheck is used. >>>>>>>>> >>>>>>>>> For the record, it's because we need to check if the host has DNS >>>>>>>>> installed. We need a LDAP connection to check this. >>>>>>>>> >>>>>>>>>> ipa-replica-install ~rcrit/replica-info-replica.example.com.gpg >>>>>>>>>> --skip-conncheck >>>>>>>>>> Directory Manager (existing master) password: >>>>>>>>>> >>>>>>>>>> ipa : ERROR Could not resolve hostname >>>>>>>>>> replica.example.com >>>>>>>>>> using DNS. Clients may not function properly. Please check your >>>>>>>>>> DNS >>>>>>>>>> setup. (Note that this check queries IPA DNS directly and ignores >>>>>>>>>> /etc/hosts.) >>>>>>>>>> Continue? [no]: >>>>>>>>>> >>>>>>>>>> So I guess, what are the intentions here? It is certainly better >>>>>>>>>> than >>>>>>>>>> before. >>>>>>>>>> >>>>>>>>>> rob >>>>>>>>> >>>>>>>>> If the replica is in the master's /etc/hosts, but not in DNS, the >>>>>>>>> conncheck will succeed. This check explicitly queries IPA records >>>>>>>>> only >>>>>>>>> and ignores /etc/hosts so it'll notice this case and warn. >>>>>>>>> >>>>>>>> >>>>>>>> Ok, like I said, this is better than we have. Just one nit then you >>>>>>>> get an ack: >>>>>>>> >>>>>>>> + # If remote host has DNS, check forward/reverse resolution >>>>>>>> + try: >>>>>>>> + entry = conn.find_entries(u'cn=dns', >>>>>>>> base_dn=DN(api.env.basedn)) >>>>>>>> + except errors.NotFound: >>>>>>>> >>>>>>>> u'cn=dns' should be str(constants.container_dns). >>>>>>>> >>>>>>>> rob >>>>>>> >>>>>>> This is a search filter, Petr could use the one I already have in >>>>>>> "dns.py::get_dns_masters()" function: >>>>>>> '(&(objectClass=ipaConfigObject)(cn=DNS))' >>>>>>> >>>>>>> For performance sake, I would also not search in the entire tree, >>>>>>> but >>>>>>> limit the >>>>>>> search only to: >>>>>>> >>>>>>> DN(('cn', 'masters'), ('cn', 'ipa'), ('cn', 'etc'), api.env.basedn) >>>>>>> >>>>>>> Martin >>>>>>> >>>>>> >>>>>> Attaching updated patch with Martin's suggestions. >>>>> >>>>> I think what Martin had in mind was: >>>>> >>>>> if api.Object.dnsrecord.get_dns_masters(): >>>>> ... >>>>> >>>> >>>> I didn't want to do this because api.Object.* use our global ldap2 >>>> Backend, which is hardwired to query localhost. >>>> I see now that I can hack around this, and we already do this in >>>> ipa-replica-install. >>>> I've extracted the hack and reused it to get the DNS masters. >>>> >>>> >>> >>> I can't say I'm crazy about the method name you've chosen... >>> >>> rob >> >> I intended the name as a warning to not use it unless necessary. >> >> Changed to temporary_ldap2_connection. >> > > I found a dangling reference to replman. I removed this and installation > seemed to work ok. > > --- install/tools/ipa-replica-install 2012-09-19 14:01:16.169053047 -0400 > +++ /usr/sbin/ipa-replica-install 2012-09-19 14:43:06.684917906 -0400 > @@ -564,8 +564,6 @@ > finally: > if conn and conn.isconnected(): > conn.disconnect() > - if replman and replman.conn: > - replman.conn.unbind_s() > > # Configure ntpd > if options.conf_ntp: > I never had a problem with this. How are you installing the replica? It's true that replman might be uninitialized if there's an error (though it doesn't seem related to the issue). Attached patch initializes it. -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0077-05-Check-direct-reverse-hostname-address-resolution-in-.patch Type: text/x-patch Size: 10240 bytes Desc: not available URL: From mkosek at redhat.com Thu Sep 20 08:37:26 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 20 Sep 2012 10:37:26 +0200 Subject: [Freeipa-devel] [PATCH] 311 Fix idrange plugin help In-Reply-To: <505AD4D0.7050102@redhat.com> References: <505AD3CA.2060907@redhat.com> <505AD4D0.7050102@redhat.com> Message-ID: <505AD5C6.20207@redhat.com> On 09/20/2012 10:33 AM, Petr Spacek wrote: > On 09/20/2012 10:28 AM, Martin Kosek wrote: >> range plugin was renamed to idrange. Update plugin help to reflect >> this change. > > ACK :-) > > Petr^2 Spacek > Pushed to master, ipa-3-0. Martin From mkosek at redhat.com Thu Sep 20 08:45:07 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 20 Sep 2012 10:45:07 +0200 Subject: [Freeipa-devel] [PATCH] SSHPublicKey.fingerprint_dns_sha1 should return unicode value In-Reply-To: <505AD24E.5020407@redhat.com> References: <505AD24E.5020407@redhat.com> Message-ID: <505AD793.20107@redhat.com> On 09/20/2012 10:22 AM, Jan Cholasta wrote: > Hi, > > this one-liner fixes updating DNS SSHFP records in host-mod. > > Honza > > ACK. Pushed to master, ipa-3-0. Martin From abokovoy at redhat.com Thu Sep 20 09:42:11 2012 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Thu, 20 Sep 2012 12:42:11 +0300 Subject: [Freeipa-devel] [PATCH] 0074 validate SID for trusted domain when adding/modifying ID range In-Reply-To: <505AD2E3.7090004@redhat.com> References: <20120919161910.GB24531@redhat.com> <505AD2E3.7090004@redhat.com> Message-ID: <20120920094210.GF24531@redhat.com> Hi, On Thu, 20 Sep 2012, Martin Kosek wrote: >On 09/19/2012 06:19 PM, Alexander Bokovoy wrote: >> Hi, >> >> This patch adds validation of SID for trusted domain when adding or >> modifying ID range for the domain. We only allow creating ranges for >> trusted domains when the trust is already established -- the default >> range is created automatically right after the trust is added. >> >> https://fedorahosted.org/freeipa/ticket/3087 >> > >Basic functionality looks OK, but I saw few issues with exception formatting: > >diff --git a/ipalib/plugins/idrange.py b/ipalib/plugins/idrange.py >index >efa906428aa58c670bc4af63b10c88123dda5b65..4750c1d6716bd69045d53f32ae1836f44e70b03b >100644 >--- a/ipalib/plugins/idrange.py >+++ b/ipalib/plugins/idrange.py >@@ -26,6 +26,12 @@ from ipapython import ipautil > from ipalib import util > from ipapython.dn import DN > >+if api.env.in_server and api.env.context in ['lite', 'server']: >+ try: >+ import ipaserver.dcerpc >+ _dcerpc_bindings_installed = True >+ except Exception, e: >+ _dcerpc_bindings_installed = False > > >Variable "e" is not used, so it can be removed. Then Exception, e should be omitted completely :) > > __doc__ = _(""" > ID ranges >@@ -137,6 +143,21 @@ user. RIDs are unique in a domain, 32bit values and are >used for users and > groups. > """) > >+def validate_trusted_domain_sid(self, sid): > >"self" is not needed in the list of attributes, this is not a class method. I'm using self.api inside. While api is singleton and accessible globally, I'd prefer passing it explicitly. So may be I'll change that to 'api'. >+ if not _dcerpc_bindings_installed: >+ raise errors.NotFound(name=_('ID Range setup'), >+ reason=_('''Cannot perform SID validation without Samba 4 >support installed. >+ Make sure you have installed server-trust-ad >sub-package of IPA on the server''')) > >Improperly formatted exception: >1) NotFound error does not use "name" param, maybe you wanted to use >ValidationError? >2) The text will be improperly formatted - since you used '''''', the >indentation will be in text: > >ipa: ERROR: Cannot perform SID validation without Samba 4 support installed. > Make sure you have installed server-trust-ad >sub-package of IPA on the server Will fix it. > >Also, I know this was discussed before, but using gettext in a name attribute >of ValidationError will cause improperly formatted exception: > ># ipa idrange-add foo --base-id=1000 --range-size=100 --dom-sid=foo >ipa: ERROR: invalid Gettext('ID Range setup', domain='ipa', localedir=None): >Options dom_sid and rid_base must be used together > >The problem is, that "name" param is printer as %r, thus you would need to >coerce it to unicode to make it better. I'd rather fix our printing code and Gettext() usage to properly give out the original string rather than swallow limitations we put on ourselves. Wider use will be, more translations will be needed and names will need to be translated as well. > >+ domain_validator = ipaserver.dcerpc.DomainValidator(self.api) >+ if not domain_validator.is_configured(): >+ raise errors.NotFound(name=_('ID Range setup'), >+ reason=_('''Cross-realm trusts are not configured.. >+ Make sure you have run ipa-adtrust-install on the >IPA server first''')) > >Same issues: > ># ipa idrange-add foo --base-id=1000 --range-size=100 --dom-sid=foo --rid-base=1000 >ipa: ERROR: Cross-realm trusts are not configured.. > Make sure you have run ipa-adtrust-install on the IPA >server first > > >+ if not domain_validator.is_trusted_sid_valid(sid): >+ raise errors.ValidationError(name=_('ID Range setup'), >+ error=_('SID is not recognized as a valid SID from a trusted >domain')) >+ >+ > >Same issues: > >ipa: ERROR: invalid Gettext('ID Range setup', domain='ipa', localedir=None): >SID is not recognized as a valid SID from a trusted domain I'll fix formatting. Thanks! -- / Alexander Bokovoy From mkosek at redhat.com Thu Sep 20 10:12:59 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 20 Sep 2012 12:12:59 +0200 Subject: [Freeipa-devel] [PATCH] 0074 validate SID for trusted domain when adding/modifying ID range In-Reply-To: <20120920094210.GF24531@redhat.com> References: <20120919161910.GB24531@redhat.com> <505AD2E3.7090004@redhat.com> <20120920094210.GF24531@redhat.com> Message-ID: <505AEC2B.80802@redhat.com> On 09/20/2012 11:42 AM, Alexander Bokovoy wrote: > Hi, > > On Thu, 20 Sep 2012, Martin Kosek wrote: >> On 09/19/2012 06:19 PM, Alexander Bokovoy wrote: >>> Hi, >>> >>> This patch adds validation of SID for trusted domain when adding or >>> modifying ID range for the domain. We only allow creating ranges for >>> trusted domains when the trust is already established -- the default >>> range is created automatically right after the trust is added. >>> >>> https://fedorahosted.org/freeipa/ticket/3087 >>> >> >> Basic functionality looks OK, but I saw few issues with exception formatting: >> >> diff --git a/ipalib/plugins/idrange.py b/ipalib/plugins/idrange.py >> index >> efa906428aa58c670bc4af63b10c88123dda5b65..4750c1d6716bd69045d53f32ae1836f44e70b03b >> >> 100644 >> --- a/ipalib/plugins/idrange.py >> +++ b/ipalib/plugins/idrange.py >> @@ -26,6 +26,12 @@ from ipapython import ipautil >> from ipalib import util >> from ipapython.dn import DN >> >> +if api.env.in_server and api.env.context in ['lite', 'server']: >> + try: >> + import ipaserver.dcerpc >> + _dcerpc_bindings_installed = True >> + except Exception, e: >> + _dcerpc_bindings_installed = False >> >> >> Variable "e" is not used, so it can be removed. > Then Exception, e should be omitted completely :) As per PEP8, "except Exception:" is preffered over bare "except:" as otherwise it would also catch SystemExit or KeyboardInterrupt. > > >> >> __doc__ = _(""" >> ID ranges >> @@ -137,6 +143,21 @@ user. RIDs are unique in a domain, 32bit values and are >> used for users and >> groups. >> """) >> >> +def validate_trusted_domain_sid(self, sid): >> >> "self" is not needed in the list of attributes, this is not a class method. > I'm using self.api inside. While api is singleton and accessible > globally, I'd prefer passing it explicitly. So may be I'll change that > to 'api'. Looks like to hack to me. I would either define this function as a method of idrange class (as I did with handle_iparangetype) and then use self.api or use "api" directly as a function parameter and not make assumptions what "self" may be. > >> + if not _dcerpc_bindings_installed: >> + raise errors.NotFound(name=_('ID Range setup'), >> + reason=_('''Cannot perform SID validation without Samba 4 >> support installed. >> + Make sure you have installed server-trust-ad >> sub-package of IPA on the server''')) >> >> Improperly formatted exception: >> 1) NotFound error does not use "name" param, maybe you wanted to use >> ValidationError? >> 2) The text will be improperly formatted - since you used '''''', the >> indentation will be in text: >> >> ipa: ERROR: Cannot perform SID validation without Samba 4 support installed. >> Make sure you have installed server-trust-ad >> sub-package of IPA on the server > Will fix it. > >> >> Also, I know this was discussed before, but using gettext in a name attribute >> of ValidationError will cause improperly formatted exception: >> >> # ipa idrange-add foo --base-id=1000 --range-size=100 --dom-sid=foo >> ipa: ERROR: invalid Gettext('ID Range setup', domain='ipa', localedir=None): >> Options dom_sid and rid_base must be used together >> >> The problem is, that "name" param is printer as %r, thus you would need to >> coerce it to unicode to make it better. > I'd rather fix our printing code and Gettext() usage to properly give > out the original string rather than swallow limitations we put on > ourselves. Wider use will be, more translations will be needed and names > will need to be translated as well. ValidationError has this format: _('invalid %(name)r: %(error)s'). AFAIU, name parameter was intended to contain only a name of attribute or option where the validation failed, i.e. a non-translable string. So this was the reason why we print it as "%r". This pattern is followed in the rest of the plugins. In your case, I think using name="dom_sid" would be the preferred use of ValidationError. Martin From pviktori at redhat.com Thu Sep 20 10:41:52 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Thu, 20 Sep 2012 12:41:52 +0200 Subject: [Freeipa-devel] [PATCH] 0074 validate SID for trusted domain when adding/modifying ID range In-Reply-To: <505AEC2B.80802@redhat.com> References: <20120919161910.GB24531@redhat.com> <505AD2E3.7090004@redhat.com> <20120920094210.GF24531@redhat.com> <505AEC2B.80802@redhat.com> Message-ID: <505AF2F0.50102@redhat.com> On 09/20/2012 12:12 PM, Martin Kosek wrote: > On 09/20/2012 11:42 AM, Alexander Bokovoy wrote: >> Hi, >> >> On Thu, 20 Sep 2012, Martin Kosek wrote: >>> On 09/19/2012 06:19 PM, Alexander Bokovoy wrote: >>>> Hi, >>>> >>>> This patch adds validation of SID for trusted domain when adding or >>>> modifying ID range for the domain. We only allow creating ranges for >>>> trusted domains when the trust is already established -- the default >>>> range is created automatically right after the trust is added. >>>> >>>> https://fedorahosted.org/freeipa/ticket/3087 >>>> >>> >>> Basic functionality looks OK, but I saw few issues with exception formatting: >>> >>> diff --git a/ipalib/plugins/idrange.py b/ipalib/plugins/idrange.py >>> index >>> efa906428aa58c670bc4af63b10c88123dda5b65..4750c1d6716bd69045d53f32ae1836f44e70b03b >>> >>> 100644 >>> --- a/ipalib/plugins/idrange.py >>> +++ b/ipalib/plugins/idrange.py >>> @@ -26,6 +26,12 @@ from ipapython import ipautil >>> from ipalib import util >>> from ipapython.dn import DN >>> >>> +if api.env.in_server and api.env.context in ['lite', 'server']: >>> + try: >>> + import ipaserver.dcerpc >>> + _dcerpc_bindings_installed = True >>> + except Exception, e: >>> + _dcerpc_bindings_installed = False >>> >>> >>> Variable "e" is not used, so it can be removed. >> Then Exception, e should be omitted completely :) > > As per PEP8, "except Exception:" is preffered over bare "except:" as otherwise > it would also catch SystemExit or KeyboardInterrupt. You should use the most specific exception you want to handle. In this case it's probably ImportError. >> >> >>> >>> __doc__ = _(""" >>> ID ranges >>> @@ -137,6 +143,21 @@ user. RIDs are unique in a domain, 32bit values and are >>> used for users and >>> groups. >>> """) >>> >>> +def validate_trusted_domain_sid(self, sid): >>> >>> "self" is not needed in the list of attributes, this is not a class method. >> I'm using self.api inside. While api is singleton and accessible >> globally, I'd prefer passing it explicitly. So may be I'll change that >> to 'api'. > > Looks like to hack to me. I would either define this function as a method of > idrange class (as I did with handle_iparangetype) and then use self.api or use > "api" directly as a function parameter and not make assumptions what "self" may be. > >> >>> + if not _dcerpc_bindings_installed: >>> + raise errors.NotFound(name=_('ID Range setup'), >>> + reason=_('''Cannot perform SID validation without Samba 4 >>> support installed. >>> + Make sure you have installed server-trust-ad >>> sub-package of IPA on the server''')) >>> >>> Improperly formatted exception: >>> 1) NotFound error does not use "name" param, maybe you wanted to use >>> ValidationError? >>> 2) The text will be improperly formatted - since you used '''''', the >>> indentation will be in text: >>> >>> ipa: ERROR: Cannot perform SID validation without Samba 4 support installed. >>> Make sure you have installed server-trust-ad >>> sub-package of IPA on the server >> Will fix it. >> >>> >>> Also, I know this was discussed before, but using gettext in a name attribute >>> of ValidationError will cause improperly formatted exception: >>> >>> # ipa idrange-add foo --base-id=1000 --range-size=100 --dom-sid=foo >>> ipa: ERROR: invalid Gettext('ID Range setup', domain='ipa', localedir=None): >>> Options dom_sid and rid_base must be used together >>> >>> The problem is, that "name" param is printer as %r, thus you would need to >>> coerce it to unicode to make it better. >> I'd rather fix our printing code and Gettext() usage to properly give >> out the original string rather than swallow limitations we put on >> ourselves. Wider use will be, more translations will be needed and names >> will need to be translated as well. > > ValidationError has this format: _('invalid %(name)r: %(error)s'). AFAIU, name > parameter was intended to contain only a name of attribute or option where the > validation failed, i.e. a non-translable string. So this was the reason why we > print it as "%r". This pattern is followed in the rest of the plugins. > > In your case, I think using name="dom_sid" would be the preferred use of > ValidationError. > > Martin > > _______________________________________________ > Freeipa-devel mailing list > Freeipa-devel at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-devel > -- Petr? From pviktori at redhat.com Thu Sep 20 10:48:22 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Thu, 20 Sep 2012 12:48:22 +0200 Subject: [Freeipa-devel] [PATCH] 0082 Use correct Dogtag port in ipaserver.install.certs Message-ID: <505AF476.7010503@redhat.com> Something I overlooked in the Dogtag 10 patch: On an instance upgraded from Dogtag 9 to Dogtag 10, ipa-replica-prepare used the wrong set of constants, and failed to contact the server. This patch fixes that. Additional fix for https://fedorahosted.org/freeipa/ticket/2846 -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0082-Use-correct-Dogtag-ports-in-ipaserver.install.certs.patch Type: text/x-patch Size: 1711 bytes Desc: not available URL: From jcholast at redhat.com Thu Sep 20 10:51:16 2012 From: jcholast at redhat.com (Jan Cholasta) Date: Thu, 20 Sep 2012 12:51:16 +0200 Subject: [Freeipa-devel] [PATCH] 1051 Fix CS replica management In-Reply-To: <5040F7DA.5020606@redhat.com> References: <5040F7DA.5020606@redhat.com> Message-ID: <505AF524.5050405@redhat.com> Hi, Dne 31.8.2012 19:43, Rob Crittenden napsal(a): > The naming in CS replication agreements is different from IPA > agreements, we have to live with what the create. The master side should > be on the local side, replica1, not the remote. This required reversing > a few master variables. > > Pass in the force flag to del_link. > > Do a better job of finding the agreements on each side. > > This should be ipa-csreplica-manage more in line with ipa-replica-manage. > > rob > Rob, can you please rebase the patch on top of current master? There were some dogtag 10 related changes to ipa-csreplica-manage since you posted the patch. Honza -- Jan Cholasta From pspacek at redhat.com Thu Sep 20 11:17:38 2012 From: pspacek at redhat.com (Petr Spacek) Date: Thu, 20 Sep 2012 13:17:38 +0200 Subject: [Freeipa-devel] [PATCH 0061] Add missing DNS view attach/detach to LDAP instance management code In-Reply-To: <5051C581.9060209@redhat.com> References: <5051C581.9060209@redhat.com> Message-ID: <505AFB52.8040206@redhat.com> On 09/13/2012 01:37 PM, Petr Spacek wrote: > Hello, > > Add missing DNS view attach/detach to LDAP instance management code. > This fixes race condition in BIND shutdown after SIGINT: > - failing assert caused by use-after-free in dns_zt_find(): > (((zt) != ((void *)0)) && (((const isc__magic_t *)(zt))->magic > == ((('Z') << 24 | ('T') << 16 | ('b') << 8 | ('l'))))) > > Petr^2 Spacek This patch causes deadlock in shut-down sequence in certain situations, I'm still searching for the root cause. Old comment from 2009-02-10 /* commented out for now, cause named to hang */ was correct, apparently ... -- Petr^2 Spacek From abokovoy at redhat.com Thu Sep 20 11:58:23 2012 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Thu, 20 Sep 2012 14:58:23 +0300 Subject: [Freeipa-devel] [PATCH] 0074 validate SID for trusted domain when adding/modifying ID range In-Reply-To: <505AF2F0.50102@redhat.com> References: <20120919161910.GB24531@redhat.com> <505AD2E3.7090004@redhat.com> <20120920094210.GF24531@redhat.com> <505AEC2B.80802@redhat.com> <505AF2F0.50102@redhat.com> Message-ID: <20120920115823.GA26222@redhat.com> On Thu, 20 Sep 2012, Petr Viktorin wrote: >On 09/20/2012 12:12 PM, Martin Kosek wrote: >>On 09/20/2012 11:42 AM, Alexander Bokovoy wrote: >>>Hi, >>> >>>On Thu, 20 Sep 2012, Martin Kosek wrote: >>>>On 09/19/2012 06:19 PM, Alexander Bokovoy wrote: >>>>>Hi, >>>>> >>>>>This patch adds validation of SID for trusted domain when adding or >>>>>modifying ID range for the domain. We only allow creating ranges for >>>>>trusted domains when the trust is already established -- the default >>>>>range is created automatically right after the trust is added. >>>>> >>>>>https://fedorahosted.org/freeipa/ticket/3087 >>>>> >>>> >>>>Basic functionality looks OK, but I saw few issues with exception formatting: >>>> >>>>diff --git a/ipalib/plugins/idrange.py b/ipalib/plugins/idrange.py >>>>index >>>>efa906428aa58c670bc4af63b10c88123dda5b65..4750c1d6716bd69045d53f32ae1836f44e70b03b >>>> >>>>100644 >>>>--- a/ipalib/plugins/idrange.py >>>>+++ b/ipalib/plugins/idrange.py >>>>@@ -26,6 +26,12 @@ from ipapython import ipautil >>>>from ipalib import util >>>>from ipapython.dn import DN >>>> >>>>+if api.env.in_server and api.env.context in ['lite', 'server']: >>>>+ try: >>>>+ import ipaserver.dcerpc >>>>+ _dcerpc_bindings_installed = True >>>>+ except Exception, e: >>>>+ _dcerpc_bindings_installed = False >>>> >>>> >>>>Variable "e" is not used, so it can be removed. >>>Then Exception, e should be omitted completely :) >> >>As per PEP8, "except Exception:" is preffered over bare "except:" as otherwise >>it would also catch SystemExit or KeyboardInterrupt. > >You should use the most specific exception you want to handle. In >this case it's probably ImportError. New patch is attached. -- / Alexander Bokovoy -------------- next part -------------- >From 4ddcad0b54e18339581a7aec042f42bec5bc7b48 Mon Sep 17 00:00:00 2001 From: Alexander Bokovoy Date: Wed, 19 Sep 2012 19:09:22 +0300 Subject: [PATCH 1/4] validate SID for trusted domain when adding/modifying ID range https://fedorahosted.org/freeipa/ticket/3087 --- ipalib/plugins/idrange.py | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/ipalib/plugins/idrange.py b/ipalib/plugins/idrange.py index ee50613bbaeb70aecf830ad480773a253f88a136..4ef3559aca0ef5314e44b727e97106866db94cda 100644 --- a/ipalib/plugins/idrange.py +++ b/ipalib/plugins/idrange.py @@ -26,6 +26,12 @@ from ipapython import ipautil from ipalib import util from ipapython.dn import DN +if api.env.in_server and api.env.context in ['lite', 'server']: + try: + import ipaserver.dcerpc + _dcerpc_bindings_installed = True + except ImportError: + _dcerpc_bindings_installed = False __doc__ = _(""" ID ranges @@ -249,6 +255,18 @@ class idrange(LDAPObject): error=_('range modification leaving objects with ID out ' 'of the defined range is not allowed')) + def validate_trusted_domain_sid(self, sid): + if not _dcerpc_bindings_installed: + raise errors.NotFound(reason=_('Cannot perform SID validation without Samba 4 support installed. ' + 'Make sure you have installed server-trust-ad sub-package of IPA on the server')) + domain_validator = ipaserver.dcerpc.DomainValidator(self.api) + if not domain_validator.is_configured(): + raise errors.NotFound(reason=_('Cross-realm trusts are not configured. ' + 'Make sure you have run ipa-adtrust-install on the IPA server first')) + if not domain_validator.is_trusted_sid_valid(sid): + raise errors.ValidationError(name='domain SID', + error=_('SID is not recognized as a valid SID for a trusted domain')) + class idrange_add(LDAPCreate): __doc__ = _(""" Add new ID range. @@ -287,6 +305,9 @@ class idrange_add(LDAPCreate): error=_('Options dom_sid and rid_base must ' \ 'be used together')) + # Validate SID as the one of trusted domains + self.obj.validate_trusted_domain_sid(options['ipanttrusteddomainsid']) + # Finally, add trusted AD domain range object class entry_attrs['objectclass'].append('ipatrustedaddomainrange') else: if (('ipasecondarybaserid' in options) != ('ipabaserid' in options)): @@ -366,6 +387,10 @@ class idrange_mod(LDAPUpdate): except errors.NotFound: self.obj.handle_not_found(*keys) + if 'ipanttrusteddomainsid' in options: + # Validate SID as the one of trusted domains + self.obj.validate_trusted_domain_sid(options['ipanttrusteddomainsid']) + old_base_id = int(old_attrs.get('ipabaseid', [0])[0]) old_range_size = int(old_attrs.get('ipaidrangesize', [0])[0]) new_base_id = entry_attrs.get('ipabaseid') -- 1.7.12 From abokovoy at redhat.com Thu Sep 20 12:01:07 2012 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Thu, 20 Sep 2012 15:01:07 +0300 Subject: [Freeipa-devel] [PATCH] 0075 Fix error messages and use proper ImportError for dcerpc Message-ID: <20120920120107.GB26222@redhat.com> Hi, fix use of NotFound error exception in plugins/group.py similar to what is discussed in patch 0074 for idrange plugin. -- / Alexander Bokovoy -------------- next part -------------- >From 9028d2c8c8da9bc259e00250093352085317f91c Mon Sep 17 00:00:00 2001 From: Alexander Bokovoy Date: Thu, 20 Sep 2012 14:02:15 +0300 Subject: [PATCH 2/4] Fix error messages and use proper ImportError for dcerpc import --- ipalib/plugins/group.py | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/ipalib/plugins/group.py b/ipalib/plugins/group.py index 011587206d809cf7ab32308f8d70eba06298c555..ae00aa8ac7d087befa5107df4eb978f1ada00240 100644 --- a/ipalib/plugins/group.py +++ b/ipalib/plugins/group.py @@ -26,7 +26,7 @@ if api.env.in_server and api.env.context in ['lite', 'server']: try: import ipaserver.dcerpc _dcerpc_bindings_installed = True - except Exception, e: + except ImportError: _dcerpc_bindings_installed = False __doc__ = _(""" @@ -328,14 +328,13 @@ class group_add_member(LDAPAddMember): result = (completed, dn) if 'ipaexternalmember' in options: if not _dcerpc_bindings_installed: - raise errors.NotFound(name=_('AD Trust'), - reason=_('''Cannot perform external member validation without Samba 4 support installed. - Make sure you have installed server-trust-ad sub-package of IPA on the server''')) + raise errors.NotFound(reason=_('Cannot perform external member validation without ' + 'Samba 4 support installed. Make sure you have installed ' + 'server-trust-ad sub-package of IPA on the server')) domain_validator = ipaserver.dcerpc.DomainValidator(self.api) if not domain_validator.is_configured(): - raise errors.NotFound(name=_('AD Trust setup'), - reason=_('''Cannot perform join operation without own domain configured. - Make sure you have run ipa-adtrust-install on the IPA server first''')) + raise errors.NotFound(reason=_('Cannot perform join operation without own domain configured. ' + 'Make sure you have run ipa-adtrust-install on the IPA server first')) sids = [] failed_sids = [] for sid in options['ipaexternalmember']: -- 1.7.12 From mkosek at redhat.com Thu Sep 20 12:02:20 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 20 Sep 2012 14:02:20 +0200 Subject: [Freeipa-devel] [PATCH] 0082 Use correct Dogtag port in ipaserver.install.certs In-Reply-To: <505AF476.7010503@redhat.com> References: <505AF476.7010503@redhat.com> Message-ID: <505B05CC.8050903@redhat.com> On 09/20/2012 12:48 PM, Petr Viktorin wrote: > Something I overlooked in the Dogtag 10 patch: > > On an instance upgraded from Dogtag 9 to Dogtag 10, ipa-replica-prepare used > the wrong set of constants, and failed to contact the server. This patch fixes > that. > > > Additional fix for https://fedorahosted.org/freeipa/ticket/2846 > Cool, fixed ipa-replica-prepare on my upgraded instance. ACK. Pushed to master, ipa-3-0. Martin From abokovoy at redhat.com Thu Sep 20 12:03:06 2012 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Thu, 20 Sep 2012 15:03:06 +0300 Subject: [Freeipa-devel] [PATCH] 0076-0077 Document trust commands and external group member Message-ID: <20120920120306.GC26222@redhat.com> Hi, attached patches 0076 and 0077 add base documentation about trust commands. Part of that documentation is also added to group membership plugin to describe external groups and external members. -- / Alexander Bokovoy -------------- next part -------------- >From bb0c11364826c0738ab7bd649101cdaeaa0081f4 Mon Sep 17 00:00:00 2001 From: Alexander Bokovoy Date: Thu, 20 Sep 2012 14:25:05 +0300 Subject: [PATCH 3/4] Add documentation for 'ipa trust' set of commands --- ipalib/plugins/trust.py | 60 +++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 58 insertions(+), 2 deletions(-) diff --git a/ipalib/plugins/trust.py b/ipalib/plugins/trust.py index bced06f4db83b98f16e75b63ba0c0c252a12e489..9d3e9a873e8f6335c12729e9f9475e59499fb3d4 100644 --- a/ipalib/plugins/trust.py +++ b/ipalib/plugins/trust.py @@ -34,11 +34,67 @@ if api.env.in_server and api.env.context in ['lite', 'server']: try: import ipaserver.dcerpc #pylint: disable=F0401 _bindings_installed = True - except Exception, e: + except ImportError: _bindings_installed = False __doc__ = _(""" -Manage trust relationship between realms +Cross-realm trusts + +Manage trust relationship between IPA and Active Directory domains. + +In order to allow users from a remote domain to access resources in IPA +domain, trust relationship needs to be established. Currently IPA supports +only trusts between IPA and Active Directory domains under control of Windows +Server 2008 or later, with functional level 2008 or later. + +Please note that DNS on both IPA and Active Directory domain sides should be +configured properly to discover each other. Trust relationship relies on +ability to discover special resources in the other domain via DNS records. + +Examples: + +1. Establish cross-realm trust with Active Directory using AD administrator + credentials: + + ipa trust-add --type=ad --admin --password + +2. List all existing trust relationships: + + ipa trust-find + +3. Show details of the specific trust relationship: + + ipa trust-show + +4. Delete existing trust relationship: + + ipa trust-del + +Once trust relationship is established, remote users will need to be mapped +to local POSIX groups in order to actually use IPA resources. The mapping should +be done via use of external membership of non-POSIX group and then this group +should be included into one of local POSIX groups. + +Example: + +1. Make note of the trusted domain security identifier + + domainsid = `ipa trust-show | grep Identifier | cut -d: -f2` + +2. Create group for the trusted domain admins' mapping and their local POSIX group: + + ipa group-add --desc=' admins external map' ad_admins_external --external + ipa group-add --desc=' admins' ad_admins + +3. Add security identifier of Domain Admins of the to the ad_admins_external + group (security identifier of -513 is Domain Admins group): + + ipa group-add-member ad_admins_external --external ${domainsid}-513 + +4. Allow members of ad_admins_external group to be associated with ad_admins POSIX group: + + ipa group-add-member ad_admins --groups ad_admins_external + """) trust_output_params = ( -- 1.7.12 -------------- next part -------------- >From 29598d8e958e571fcba0c4a81ea671092375b727 Mon Sep 17 00:00:00 2001 From: Alexander Bokovoy Date: Thu, 20 Sep 2012 14:31:01 +0300 Subject: [PATCH 4/4] Document use of external group membership --- ipalib/plugins/group.py | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/ipalib/plugins/group.py b/ipalib/plugins/group.py index ae00aa8ac7d087befa5107df4eb978f1ada00240..3775056a12400ddc236bf5c12ff862731f699431 100644 --- a/ipalib/plugins/group.py +++ b/ipalib/plugins/group.py @@ -76,6 +76,35 @@ EXAMPLES: Display information about a named group. ipa group-show localadmins + +External group membership is designed to allow users from trusted domains +to be mapped to local POSIX groups in order to actually use IPA resources. +External members should be added to groups that specifically created as +external and non-POSIX. Such group later should be included into one of POSIX +groups. + +An external group member is currently a Security Identifier as defined by +the trusted domain. + +Example: + +1. Make note of the trusted domain security identifier + + domainsid = `ipa trust-show | grep Identifier | cut -d: -f2` + +2. Create group for the trusted domain admins' mapping and their local POSIX group: + + ipa group-add --desc=' admins external map' ad_admins_external --external + ipa group-add --desc=' admins' ad_admins + +3. Add security identifier of Domain Admins of the to the ad_admins_external + group (security identifier of -513 is Domain Admins group): + + ipa group-add-member ad_admins_external --external ${domainsid}-513 + +4. Allow members of ad_admins_external group to be associated with ad_admins POSIX group: + + ipa group-add-member ad_admins --groups ad_admins_external """) protected_group_name = u'admins' -- 1.7.12 From mkosek at redhat.com Thu Sep 20 12:20:14 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 20 Sep 2012 14:20:14 +0200 Subject: [Freeipa-devel] [PATCH] 0074 validate SID for trusted domain when adding/modifying ID range In-Reply-To: <20120920115823.GA26222@redhat.com> References: <20120919161910.GB24531@redhat.com> <505AD2E3.7090004@redhat.com> <20120920094210.GF24531@redhat.com> <505AEC2B.80802@redhat.com> <505AF2F0.50102@redhat.com> <20120920115823.GA26222@redhat.com> Message-ID: <505B09FE.6090406@redhat.com> On 09/20/2012 01:58 PM, Alexander Bokovoy wrote: > On Thu, 20 Sep 2012, Petr Viktorin wrote: >> On 09/20/2012 12:12 PM, Martin Kosek wrote: >>> On 09/20/2012 11:42 AM, Alexander Bokovoy wrote: >>>> Hi, >>>> >>>> On Thu, 20 Sep 2012, Martin Kosek wrote: >>>>> On 09/19/2012 06:19 PM, Alexander Bokovoy wrote: >>>>>> Hi, >>>>>> >>>>>> This patch adds validation of SID for trusted domain when adding or >>>>>> modifying ID range for the domain. We only allow creating ranges for >>>>>> trusted domains when the trust is already established -- the default >>>>>> range is created automatically right after the trust is added. >>>>>> >>>>>> https://fedorahosted.org/freeipa/ticket/3087 >>>>>> >>>>> >>>>> Basic functionality looks OK, but I saw few issues with exception formatting: >>>>> >>>>> diff --git a/ipalib/plugins/idrange.py b/ipalib/plugins/idrange.py >>>>> index >>>>> efa906428aa58c670bc4af63b10c88123dda5b65..4750c1d6716bd69045d53f32ae1836f44e70b03b >>>>> >>>>> >>>>> 100644 >>>>> --- a/ipalib/plugins/idrange.py >>>>> +++ b/ipalib/plugins/idrange.py >>>>> @@ -26,6 +26,12 @@ from ipapython import ipautil >>>>> from ipalib import util >>>>> from ipapython.dn import DN >>>>> >>>>> +if api.env.in_server and api.env.context in ['lite', 'server']: >>>>> + try: >>>>> + import ipaserver.dcerpc >>>>> + _dcerpc_bindings_installed = True >>>>> + except Exception, e: >>>>> + _dcerpc_bindings_installed = False >>>>> >>>>> >>>>> Variable "e" is not used, so it can be removed. >>>> Then Exception, e should be omitted completely :) >>> >>> As per PEP8, "except Exception:" is preffered over bare "except:" as otherwise >>> it would also catch SystemExit or KeyboardInterrupt. >> >> You should use the most specific exception you want to handle. In this case >> it's probably ImportError. > New patch is attached. > The patch looks OK, I would just also like to have the rest of the name=_('ID Range setup') messages fixed so that we don't print confusing errors: $ git grep "ID Range setup" ipalib/ ipalib/plugins/idrange.py: raise errors.ValidationError(name=_('ID Range setup'), ipalib/plugins/idrange.py: raise errors.ValidationError(name=_('ID Range setup'), ipalib/plugins/idrange.py: raise errors.ValidationError(name=_('ID Range setup'), Martin From abokovoy at redhat.com Thu Sep 20 12:31:24 2012 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Thu, 20 Sep 2012 15:31:24 +0300 Subject: [Freeipa-devel] [PATCH] 0074 validate SID for trusted domain when adding/modifying ID range In-Reply-To: <505B09FE.6090406@redhat.com> References: <20120919161910.GB24531@redhat.com> <505AD2E3.7090004@redhat.com> <20120920094210.GF24531@redhat.com> <505AEC2B.80802@redhat.com> <505AF2F0.50102@redhat.com> <20120920115823.GA26222@redhat.com> <505B09FE.6090406@redhat.com> Message-ID: <20120920123124.GD26222@redhat.com> On Thu, 20 Sep 2012, Martin Kosek wrote: >On 09/20/2012 01:58 PM, Alexander Bokovoy wrote: >> On Thu, 20 Sep 2012, Petr Viktorin wrote: >>> On 09/20/2012 12:12 PM, Martin Kosek wrote: >>>> On 09/20/2012 11:42 AM, Alexander Bokovoy wrote: >>>>> Hi, >>>>> >>>>> On Thu, 20 Sep 2012, Martin Kosek wrote: >>>>>> On 09/19/2012 06:19 PM, Alexander Bokovoy wrote: >>>>>>> Hi, >>>>>>> >>>>>>> This patch adds validation of SID for trusted domain when adding or >>>>>>> modifying ID range for the domain. We only allow creating ranges for >>>>>>> trusted domains when the trust is already established -- the default >>>>>>> range is created automatically right after the trust is added. >>>>>>> >>>>>>> https://fedorahosted.org/freeipa/ticket/3087 >>>>>>> >>>>>> >>>>>> Basic functionality looks OK, but I saw few issues with exception formatting: >>>>>> >>>>>> diff --git a/ipalib/plugins/idrange.py b/ipalib/plugins/idrange.py >>>>>> index >>>>>> efa906428aa58c670bc4af63b10c88123dda5b65..4750c1d6716bd69045d53f32ae1836f44e70b03b >>>>>> >>>>>> >>>>>> 100644 >>>>>> --- a/ipalib/plugins/idrange.py >>>>>> +++ b/ipalib/plugins/idrange.py >>>>>> @@ -26,6 +26,12 @@ from ipapython import ipautil >>>>>> from ipalib import util >>>>>> from ipapython.dn import DN >>>>>> >>>>>> +if api.env.in_server and api.env.context in ['lite', 'server']: >>>>>> + try: >>>>>> + import ipaserver.dcerpc >>>>>> + _dcerpc_bindings_installed = True >>>>>> + except Exception, e: >>>>>> + _dcerpc_bindings_installed = False >>>>>> >>>>>> >>>>>> Variable "e" is not used, so it can be removed. >>>>> Then Exception, e should be omitted completely :) >>>> >>>> As per PEP8, "except Exception:" is preffered over bare "except:" as otherwise >>>> it would also catch SystemExit or KeyboardInterrupt. >>> >>> You should use the most specific exception you want to handle. In this case >>> it's probably ImportError. >> New patch is attached. >> > >The patch looks OK, I would just also like to have the rest of the name=_('ID >Range setup') messages fixed so that we don't print confusing errors: > >$ git grep "ID Range setup" ipalib/ >ipalib/plugins/idrange.py: raise >errors.ValidationError(name=_('ID Range setup'), >ipalib/plugins/idrange.py: raise >errors.ValidationError(name=_('ID Range setup'), >ipalib/plugins/idrange.py: raise >errors.ValidationError(name=_('ID Range setup'), Yep. Done. -- / Alexander Bokovoy -------------- next part -------------- >From 86fef992edd0f6e7b5c818c774352067e90e02a3 Mon Sep 17 00:00:00 2001 From: Alexander Bokovoy Date: Wed, 19 Sep 2012 19:09:22 +0300 Subject: [PATCH 1/4] validate SID for trusted domain when adding/modifying ID range https://fedorahosted.org/freeipa/ticket/3087 --- ipalib/plugins/idrange.py | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-) diff --git a/ipalib/plugins/idrange.py b/ipalib/plugins/idrange.py index ee50613bbaeb70aecf830ad480773a253f88a136..8f2d4efdc0463e7d81cd72fba7769e38dc0c638b 100644 --- a/ipalib/plugins/idrange.py +++ b/ipalib/plugins/idrange.py @@ -26,6 +26,12 @@ from ipapython import ipautil from ipalib import util from ipapython.dn import DN +if api.env.in_server and api.env.context in ['lite', 'server']: + try: + import ipaserver.dcerpc + _dcerpc_bindings_installed = True + except ImportError: + _dcerpc_bindings_installed = False __doc__ = _(""" ID ranges @@ -249,6 +255,18 @@ class idrange(LDAPObject): error=_('range modification leaving objects with ID out ' 'of the defined range is not allowed')) + def validate_trusted_domain_sid(self, sid): + if not _dcerpc_bindings_installed: + raise errors.NotFound(reason=_('Cannot perform SID validation without Samba 4 support installed. ' + 'Make sure you have installed server-trust-ad sub-package of IPA on the server')) + domain_validator = ipaserver.dcerpc.DomainValidator(self.api) + if not domain_validator.is_configured(): + raise errors.NotFound(reason=_('Cross-realm trusts are not configured. ' + 'Make sure you have run ipa-adtrust-install on the IPA server first')) + if not domain_validator.is_trusted_sid_valid(sid): + raise errors.ValidationError(name='domain SID', + error=_('SID is not recognized as a valid SID for a trusted domain')) + class idrange_add(LDAPCreate): __doc__ = _(""" Add new ID range. @@ -278,19 +296,22 @@ class idrange_add(LDAPCreate): if 'ipanttrusteddomainsid' in options: if 'ipasecondarybaserid' in options: - raise errors.ValidationError(name=_('ID Range setup'), + raise errors.ValidationError(name='ID Range setup', error=_('Options dom_sid and secondary_rid_base cannot ' \ 'be used together')) if 'ipabaserid' not in options: - raise errors.ValidationError(name=_('ID Range setup'), + raise errors.ValidationError(name='ID Range setup', error=_('Options dom_sid and rid_base must ' \ 'be used together')) + # Validate SID as the one of trusted domains + self.obj.validate_trusted_domain_sid(options['ipanttrusteddomainsid']) + # Finally, add trusted AD domain range object class entry_attrs['objectclass'].append('ipatrustedaddomainrange') else: if (('ipasecondarybaserid' in options) != ('ipabaserid' in options)): - raise errors.ValidationError(name=_('ID Range setup'), + raise errors.ValidationError(name='ID Range setup', error=_('Options secondary_rid_base and rid_base must ' \ 'be used together')) @@ -366,6 +387,10 @@ class idrange_mod(LDAPUpdate): except errors.NotFound: self.obj.handle_not_found(*keys) + if 'ipanttrusteddomainsid' in options: + # Validate SID as the one of trusted domains + self.obj.validate_trusted_domain_sid(options['ipanttrusteddomainsid']) + old_base_id = int(old_attrs.get('ipabaseid', [0])[0]) old_range_size = int(old_attrs.get('ipaidrangesize', [0])[0]) new_base_id = entry_attrs.get('ipabaseid') -- 1.7.12 From mkosek at redhat.com Thu Sep 20 12:42:35 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 20 Sep 2012 14:42:35 +0200 Subject: [Freeipa-devel] [PATCH 0006] Improves sssd.conf handling during ipa-client uninstall In-Reply-To: <50583D07.6040001@redhat.com> References: <717623750.1969105.1345208663118.JavaMail.root@redhat.com> <503B5D09.7040401@redhat.com> <503B6EC8.3060508@redhat.com> <20120827133727.GG8612@zeppelin.brq.redhat.com> <503B8A65.30100@redhat.com> <503E1114.3080309@redhat.com> <5050AA68.2010300@redhat.com> <50583D07.6040001@redhat.com> Message-ID: <505B0F3B.3010704@redhat.com> On 09/18/2012 11:21 AM, Tomas Babej wrote: > On 09/12/2012 05:29 PM, Martin Kosek wrote: >> On 08/29/2012 02:54 PM, Tomas Babej wrote: >>> On 08/27/2012 04:55 PM, Martin Kosek wrote: >>>> On 08/27/2012 03:37 PM, Jakub Hrozek wrote: >>>>> On Mon, Aug 27, 2012 at 02:57:44PM +0200, Martin Kosek wrote: >>>>>> I think that the right behavior of SSSD conf uninstall should be the >>>>>> following: >>>>>> >>>>>> * sssd.conf existed before IPA install + non-IPA domains in sssd.conf found: >>>>>> - move backed conf up sssd.conf.bkp (and inform the user) >>>>>> - use SSSDConfig delete_domain function to remove ipa domain from >>>>>> sssd.conf >>>>>> - restart sssd afterwards >>>>> I'm confused here, which of the files is the original >>>>> pre-ipa-client-install file? >>>> This is the "backed up sssd.conf". I thought that it may be useful for user to >>>> still have an access to it after uninstall. >>>> >>>>> How does the non-ipa domain end up in the sssd.conf file? Does it have >>>>> to be configured manually or does ipa-client-install merge the list of >>>>> domains on installation? >>>> ipa-client-install merge the list of the domains. It overrides the old >>>> sssd.conf only when we cannot parse the sssd.conf and --preserve-sssd option >>>> was not set. >>>> >>>> Martin >>> Hi, >>> >>> The sssd.conf file is no longer left behind in case sssd was not >>> configured before the installation. However, the patch goes behind >>> the scope of this ticked and improves the handling of sssd.conf >>> during the ipa-client-install --uninstall in general. >>> >>> The current behaviour (well documented in source code) is as follows: >>> - In general, the IPA domain is simply removed from the sssd.conf >>> file, instead of sssd.conf being rewritten from the backup. This >>> preserves any domains added after installation. >>> >>> - If sssd.conf existed before the installation, it is restored to >>> sssd.conf.bkp. However, any IPA domains from pre-installation >>> sssd.conf should have been merged during the installation. >>> >>> - If sssd.conf did not exist before the installation, and no other >>> domains than IPA domain exist in it, the patch makes sure that >>> sssd.conf is moved to sssd.conf.deleted so user experiences no >>> crash during any next installation due to its existence. >>> >>> https://fedorahosted.org/freeipa/ticket/2740 >>> >>> Tomas >>> >> Good job, SSSD uninstall process now looks more consistent and better >> documented. I just found the following (mainly minor) issues. Comments in the >> patch: >> >> diff --git a/ipa-client/ipa-install/ipa-client-install >> b/ipa-client/ipa-install/ipa-client-install >> index >> 2e65921e8de2dfe68443f5b5875954d71dd48ed2..c5cef15e1fb3a3e1d7cfd070f4288d3839accfc8 >> >> 100755 >> --- a/ipa-client/ipa-install/ipa-client-install >> +++ b/ipa-client/ipa-install/ipa-client-install >> @@ -183,6 +183,36 @@ def nssldap_exists(): >> >> return (retval, files_found) >> >> +# helper function for uninstall >> +# deletes IPA domain from sssd.conf >> +def delete_IPA_domain(): >> >> Function names should be lowercase -> delete_ipa_domain >> >> + sssd = ipaservices.service('sssd') >> + try: >> + sssdconfig = SSSDConfig.SSSDConfig() >> + sssdconfig.import_config() >> + domains = sssdconfig.list_active_domains() >> + >> + IPA_domain_name = None >> >> Variables should be lowercase -> ipa_domain_name >> >> + >> + for name in domains: >> + domain = sssdconfig.get_domain(name) >> + try: >> + provider = domain.get_option('id_provider') >> + if provider == "ipa": >> + IPA_domain_name = name >> + break >> + except SSSDConfig.NoOptionError: >> + continue >> + >> + if IPA_domain_name != None: >> >> Do not use != with None, True, False - use "is not None". >> >> + sssdconfig.delete_domain(IPA_domain_name) >> + sssdconfig.write() >> + else: >> + root_logger.warning("IPA domain could not be found in " + >> + "sssd.conf and therefore not deleted") >> + except IOError: >> + root_logger.warning("IPA domain could not be deleted. No access to the >> sssd.conf file.") >> >> There should be full path to sssd.conf in this error message. It is very useful >> sometimes. >> >> + >> def uninstall(options, env): >> >> if not fstore.has_files(): >> @@ -212,7 +242,12 @@ def uninstall(options, env): >> sssdconfig = SSSDConfig.SSSDConfig() >> sssdconfig.import_config() >> domains = sssdconfig.list_active_domains() >> - if len(domains) > 1: >> + all_domains = sssdconfig.list_domains() >> + >> + # we consider all the domains, because handling sssd.conf >> + # during uninstall is dependant on was_sssd_configured flag >> + # so the user does not lose info about inactive domains >> + if len(all_domains) > 1: >> # There was more than IPA domain configured >> was_sssd_configured = True >> for name in domains: >> @@ -349,6 +384,62 @@ def uninstall(options, env): >> "Failed to remove krb5/LDAP configuration: %s", str(e)) >> return CLIENT_INSTALL_ERROR >> >> + # Next if-elif-elif construction deals with sssd.conf file. >> + # Old pre-IPA domains are preserved due merging the old sssd.conf >> + # during the installation of ipa-client but any new domains are >> + # only present in sssd.conf now, so we don't want to delete them >> + # by rewriting sssd.conf file. IPA domain is removed gracefully. >> + >> + # SSSD was installed before our installation and other non-IPA domains >> + # found, restore backed up sssd.conf to sssd.conf.bkp and remove IPA >> + # domain from the current sssd.conf >> + if was_sssd_installed and was_sssd_configured: >> + root_logger.info( >> + "The original configuration of SSSD included other domains than " + >> + "the IPA-based one.") >> + >> + delete_IPA_domain() >> + >> >> + try: >> + os.rename("/etc/sssd/sssd.conf","/etc/sssd/sssd.conf.current") >> + if fstore.restore_file("/etc/sssd/sssd.conf"): >> + os.rename("/etc/sssd/sssd.conf","/etc/sssd/sssd.conf.bkp") >> + os.rename("/etc/sssd/sssd.conf.current","/etc/sssd/sssd.conf") >> >> I don't like this very much. I would rather not mess with /etc/sssd/sssd.conf, >> it may surprise other tools/user and we could also end with no >> /etc/sssd/sssd.conf at all if OSError is raised in second or third step. >> >> I would rather enhance fstore and implement function like "restore_file_to" >> which would accept a second parameter with a custom path, like >> "/etc/sssd/sssd.conf.bkp". >> >> + except OSError: >> + pass >> >> The exception should be at least logged to debug log. >> >> + >> + root_logger.info("Original pre-IPA SSSD configuration file was " + >> + "restored to /etc/sssd/sssd.conf.bkp.") >> + root_logger.info("IPA domain removed from current one, " + >> + "restarting SSSD service") >> + sssd = ipaservices.service('sssd') >> + try: >> + sssd.restart() >> + except CalledProcessError: >> + root_logger.warning("SSSD service restart was unsuccessful.") >> + >> + # SSSD was not installed before our installation, but other domains found, >> + # delete IPA domain, but leave other domains intact >> + elif not was_sssd_installed and was_sssd_configured: >> + delete_IPA_domain() >> + root_logger.info("Other domains than IPA domain found, " + >> + "IPA domain was removed from sssd.conf.") >> + try: >> + sssd.restart() >> + except CalledProcessError: >> + root_logger.warning("SSSD service restart was unsuccessful.") >> + >> + # SSSD was not installed before our installation, and no other domains >> + # than IPA are configured in sssd.conf - make sure config file is removed >> + elif not was_sssd_installed and not was_sssd_configured: >> + try: >> + os.rename("/etc/sssd/sssd.conf","/etc/sssd/sssd.conf.deleted") >> + except OSError: >> + pass >> >> >> Error should be logged to debug log. >> >> + >> + root_logger.info("Redundant SSSD configuration file " + >> + "/etc/sssd/sssd.conf was moved to /etc/sssd.conf.deleted") >> + >> if fstore.has_files(): >> root_logger.info("Restoring client configuration files") >> fstore.restore_all_files() >> @@ -428,20 +519,6 @@ def uninstall(options, env): >> if was_sshd_configured and ipaservices.knownservices.sshd.is_running(): >> ipaservices.knownservices.sshd.restart() >> >> - if was_sssd_installed and was_sssd_configured: >> - # SSSD was installed before our installation, config now is restored, >> restart it >> - root_logger.info( >> - "The original configuration of SSSD included other domains than " + >> - "the IPA-based one.") >> - root_logger.info( >> - "Original configuration file was restored, restarting SSSD " + >> - "service.") >> - sssd = ipaservices.service('sssd') >> - try: >> - sssd.restart() >> - except CalledProcessError: >> - root_logger.warning("SSSD service restart was unsuccessful.") >> - >> if not options.unattended: >> root_logger.info( >> "The original nsswitch.conf configuration has been restored.") > Final (hopefully) version attached. > > Tomas Well... almost the final version :-) 1) Wrong path to sssd.conf.deleted: #Unenrolling client from IPA server Removing Kerberos service principals from /etc/krb5.keytab Disabling client Kerberos and LDAP configurations Redundant SSSD configuration file /etc/sssd/sssd.conf was moved to /etc/sssd.conf.deleted <<<<<<<< Restoring client configuration files Client uninstall complete. ipa-client-install --uninstall --unattended 2) Uninstall crashes in the second case: # ipa-client-install --uninstall --unattended Unenrolling client from IPA server Removing Kerberos service principals from /etc/krb5.keytab Disabling client Kerberos and LDAP configurations Other domains than IPA domain found, IPA domain was removed from sssd.conf. Traceback (most recent call last): File "/sbin/ipa-client-install", line 1913, in sys.exit(main()) File "/sbin/ipa-client-install", line 1891, in main return uninstall(options, env) File "/sbin/ipa-client-install", line 432, in uninstall sssd.restart() UnboundLocalError: local variable 'sssd' referenced before assignment Also, full path to sssd.conf in the info message would be fine. Otherwise, the patch seems to work fine. Martin From mkosek at redhat.com Thu Sep 20 12:56:55 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 20 Sep 2012 14:56:55 +0200 Subject: [Freeipa-devel] [PATCH] 0074 validate SID for trusted domain when adding/modifying ID range In-Reply-To: <20120920123124.GD26222@redhat.com> References: <20120919161910.GB24531@redhat.com> <505AD2E3.7090004@redhat.com> <20120920094210.GF24531@redhat.com> <505AEC2B.80802@redhat.com> <505AF2F0.50102@redhat.com> <20120920115823.GA26222@redhat.com> <505B09FE.6090406@redhat.com> <20120920123124.GD26222@redhat.com> Message-ID: <505B1297.6070908@redhat.com> On 09/20/2012 02:31 PM, Alexander Bokovoy wrote: > On Thu, 20 Sep 2012, Martin Kosek wrote: >> On 09/20/2012 01:58 PM, Alexander Bokovoy wrote: >>> On Thu, 20 Sep 2012, Petr Viktorin wrote: >>>> On 09/20/2012 12:12 PM, Martin Kosek wrote: >>>>> On 09/20/2012 11:42 AM, Alexander Bokovoy wrote: >>>>>> Hi, >>>>>> >>>>>> On Thu, 20 Sep 2012, Martin Kosek wrote: >>>>>>> On 09/19/2012 06:19 PM, Alexander Bokovoy wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> This patch adds validation of SID for trusted domain when adding or >>>>>>>> modifying ID range for the domain. We only allow creating ranges for >>>>>>>> trusted domains when the trust is already established -- the default >>>>>>>> range is created automatically right after the trust is added. >>>>>>>> >>>>>>>> https://fedorahosted.org/freeipa/ticket/3087 >>>>>>>> >>>>>>> >>>>>>> Basic functionality looks OK, but I saw few issues with exception >>>>>>> formatting: >>>>>>> >>>>>>> diff --git a/ipalib/plugins/idrange.py b/ipalib/plugins/idrange.py >>>>>>> index >>>>>>> efa906428aa58c670bc4af63b10c88123dda5b65..4750c1d6716bd69045d53f32ae1836f44e70b03b >>>>>>> >>>>>>> >>>>>>> >>>>>>> 100644 >>>>>>> --- a/ipalib/plugins/idrange.py >>>>>>> +++ b/ipalib/plugins/idrange.py >>>>>>> @@ -26,6 +26,12 @@ from ipapython import ipautil >>>>>>> from ipalib import util >>>>>>> from ipapython.dn import DN >>>>>>> >>>>>>> +if api.env.in_server and api.env.context in ['lite', 'server']: >>>>>>> + try: >>>>>>> + import ipaserver.dcerpc >>>>>>> + _dcerpc_bindings_installed = True >>>>>>> + except Exception, e: >>>>>>> + _dcerpc_bindings_installed = False >>>>>>> >>>>>>> >>>>>>> Variable "e" is not used, so it can be removed. >>>>>> Then Exception, e should be omitted completely :) >>>>> >>>>> As per PEP8, "except Exception:" is preffered over bare "except:" as >>>>> otherwise >>>>> it would also catch SystemExit or KeyboardInterrupt. >>>> >>>> You should use the most specific exception you want to handle. In this case >>>> it's probably ImportError. >>> New patch is attached. >>> >> >> The patch looks OK, I would just also like to have the rest of the name=_('ID >> Range setup') messages fixed so that we don't print confusing errors: >> >> $ git grep "ID Range setup" ipalib/ >> ipalib/plugins/idrange.py: raise >> errors.ValidationError(name=_('ID Range setup'), >> ipalib/plugins/idrange.py: raise >> errors.ValidationError(name=_('ID Range setup'), >> ipalib/plugins/idrange.py: raise >> errors.ValidationError(name=_('ID Range setup'), > Yep. Done. > > ACK. Pushed to master, ipa-3-0. Martin From mkosek at redhat.com Thu Sep 20 12:59:25 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 20 Sep 2012 14:59:25 +0200 Subject: [Freeipa-devel] [PATCH] 0075 Fix error messages and use proper ImportError for dcerpc In-Reply-To: <20120920120107.GB26222@redhat.com> References: <20120920120107.GB26222@redhat.com> Message-ID: <505B132D.3070708@redhat.com> On 09/20/2012 02:01 PM, Alexander Bokovoy wrote: > Hi, > > fix use of NotFound error exception in plugins/group.py similar to what > is discussed in patch 0074 for idrange plugin. > ACK. Pushed to master, ipa-3-0. Martin From mkosek at redhat.com Thu Sep 20 13:00:01 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 20 Sep 2012 15:00:01 +0200 Subject: [Freeipa-devel] [PATCH] 0076-0077 Document trust commands and external group member In-Reply-To: <20120920120306.GC26222@redhat.com> References: <20120920120306.GC26222@redhat.com> Message-ID: <505B1351.9010602@redhat.com> On 09/20/2012 02:03 PM, Alexander Bokovoy wrote: > Hi, > > attached patches 0076 and 0077 add base documentation about trust > commands. Part of that documentation is also added to group membership > plugin to describe external groups and external members. > ACK. Pushed both patches to master, ipa-3-0. Martin From mkosek at redhat.com Thu Sep 20 13:37:21 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 20 Sep 2012 15:37:21 +0200 Subject: [Freeipa-devel] [PATCH] 0081 Only stop the main DS instance when upgrading it In-Reply-To: <50534F3B.6030603@redhat.com> References: <505328D1.2020806@redhat.com> <1347628337.19861.31.camel@willson.li.ssimo.org> <50533B69.2030905@redhat.com> <505344F0.3020901@redhat.com> <50534F3B.6030603@redhat.com> Message-ID: <505B1C11.1030702@redhat.com> On 09/14/2012 05:37 PM, Martin Kosek wrote: > On 09/14/2012 04:53 PM, Petr Viktorin wrote: >> On 09/14/2012 04:12 PM, Petr Viktorin wrote: >>> On 09/14/2012 03:12 PM, Simo Sorce wrote: >>>> On Fri, 2012-09-14 at 14:53 +0200, Petr Viktorin wrote: >>>>> This fixes a 2.2?3.0 upgrade bug found while testing the Dogtag 10 work. >>>>> See commit or ticket for details. >>>>> >>>>> https://fedorahosted.org/freeipa/ticket/3083 >>>>> >>>>> >>>>> >>>>> I also suspect that waiting for ports is not a good way to check if the >>>>> CMS is fully initialized, but I don't know of a better way. If you know >>>>> one, please speak up. >>>>> >> >> Such a better way is coming with https://fedorahosted.org/pki/ticket/314. I >> opened https://fedorahosted.org/freeipa/ticket/3084 so we won't forget to take >> advantage of it. >> >>>> Won;t we get back to square zero when the work to merge DS and CS >>>> instances into one will be completed ? >>>> >>>> Simo. >>>> >>> >>> When they're merged we'll probably need to bring down the CA while >>> upgrading the server. Or just stop all the IPA services to be on the >>> safe side, and of course bring them back up afterwards. >>> Meanwhile, this patch shouldn't hurt things. > > The patch worked fine for me. It will be useful at least to the point when we > use a common DS instance, as Simo suggested. > > Martin > Just for the record - this patch was ACKed and pushed by Rob on Sep 18th to master, ipa-3-0. Martin From pspacek at redhat.com Thu Sep 20 14:16:41 2012 From: pspacek at redhat.com (Petr Spacek) Date: Thu, 20 Sep 2012 16:16:41 +0200 Subject: [Freeipa-devel] [PATCH 0065] Bump version in .spec file to 2.0 Message-ID: <505B2549.9080808@redhat.com> Hello, this patch bumps version in .spec file to 2.0. -- Petr^2 Spacek -------------- next part -------------- A non-text attachment was scrubbed... Name: bind-dyndb-ldap-pspacek-0065-Bump-version-in-.spec-file-to-2.0.patch Type: text/x-patch Size: 1037 bytes Desc: not available URL: From atkac at redhat.com Thu Sep 20 14:18:57 2012 From: atkac at redhat.com (Adam Tkac) Date: Thu, 20 Sep 2012 16:18:57 +0200 Subject: [Freeipa-devel] [PATCH 0065] Bump version in .spec file to 2.0 In-Reply-To: <505B2549.9080808@redhat.com> References: <505B2549.9080808@redhat.com> Message-ID: <20120920141856.GA23503@redhat.com> On Thu, Sep 20, 2012 at 04:16:41PM +0200, Petr Spacek wrote: > Hello, > > this patch bumps version in .spec file to 2.0. Ack > From b4fc1e119e5d602c196af47bde07d3cfe3091a3d Mon Sep 17 00:00:00 2001 > From: Petr Spacek > Date: Thu, 20 Sep 2012 16:14:05 +0200 > Subject: [PATCH] Bump version in .spec file to 2.0. > > Signed-off-by: Petr Spacek > --- > contrib/bind-dyndb-ldap.spec | 10 +++------- > 1 file changed, 3 insertions(+), 7 deletions(-) > > diff --git a/contrib/bind-dyndb-ldap.spec b/contrib/bind-dyndb-ldap.spec > index 664e6a9af9629f07193c82382debac028c425fe1..5f1e0262e4a4707a6bb5a5820f807a98390581ce 100644 > --- a/contrib/bind-dyndb-ldap.spec > +++ b/contrib/bind-dyndb-ldap.spec > @@ -1,12 +1,8 @@ > -#%define PATCHVER P4 > -%define PREVER rc1 > -#%define VERSION %{version} > -#%define VERSION %{version}-%{PATCHVER} > -%define VERSION %{version}%{PREVER} > +%define VERSION %{version} > > Name: bind-dyndb-ldap > -Version: 1.1.0 > -Release: 0.1.%{PREVER}%{?dist} > +Version: 2.0 > +Release: 0%{?dist} > Summary: LDAP back-end plug-in for BIND > > Group: System Environment/Libraries > -- > 1.7.11.4 > -- Adam Tkac, Red Hat, Inc. From tbabej at redhat.com Thu Sep 20 14:20:22 2012 From: tbabej at redhat.com (Tomas Babej) Date: Thu, 20 Sep 2012 16:20:22 +0200 Subject: [Freeipa-devel] [PATCH 0006] Improves sssd.conf handling during ipa-client uninstall In-Reply-To: <505B0F3B.3010704@redhat.com> References: <717623750.1969105.1345208663118.JavaMail.root@redhat.com> <503B5D09.7040401@redhat.com> <503B6EC8.3060508@redhat.com> <20120827133727.GG8612@zeppelin.brq.redhat.com> <503B8A65.30100@redhat.com> <503E1114.3080309@redhat.com> <5050AA68.2010300@redhat.com> <50583D07.6040001@redhat.com> <505B0F3B.3010704@redhat.com> Message-ID: <505B2626.4000801@redhat.com> On 09/20/2012 02:42 PM, Martin Kosek wrote: > On 09/18/2012 11:21 AM, Tomas Babej wrote: >> On 09/12/2012 05:29 PM, Martin Kosek wrote: >>> On 08/29/2012 02:54 PM, Tomas Babej wrote: >>>> On 08/27/2012 04:55 PM, Martin Kosek wrote: >>>>> On 08/27/2012 03:37 PM, Jakub Hrozek wrote: >>>>>> On Mon, Aug 27, 2012 at 02:57:44PM +0200, Martin Kosek wrote: >>>>>>> I think that the right behavior of SSSD conf uninstall should be the >>>>>>> following: >>>>>>> >>>>>>> * sssd.conf existed before IPA install + non-IPA domains in sssd.conf found: >>>>>>> - move backed conf up sssd.conf.bkp (and inform the user) >>>>>>> - use SSSDConfig delete_domain function to remove ipa domain from >>>>>>> sssd.conf >>>>>>> - restart sssd afterwards >>>>>> I'm confused here, which of the files is the original >>>>>> pre-ipa-client-install file? >>>>> This is the "backed up sssd.conf". I thought that it may be useful for user to >>>>> still have an access to it after uninstall. >>>>> >>>>>> How does the non-ipa domain end up in the sssd.conf file? Does it have >>>>>> to be configured manually or does ipa-client-install merge the list of >>>>>> domains on installation? >>>>> ipa-client-install merge the list of the domains. It overrides the old >>>>> sssd.conf only when we cannot parse the sssd.conf and --preserve-sssd option >>>>> was not set. >>>>> >>>>> Martin >>>> Hi, >>>> >>>> The sssd.conf file is no longer left behind in case sssd was not >>>> configured before the installation. However, the patch goes behind >>>> the scope of this ticked and improves the handling of sssd.conf >>>> during the ipa-client-install --uninstall in general. >>>> >>>> The current behaviour (well documented in source code) is as follows: >>>> - In general, the IPA domain is simply removed from the sssd.conf >>>> file, instead of sssd.conf being rewritten from the backup. This >>>> preserves any domains added after installation. >>>> >>>> - If sssd.conf existed before the installation, it is restored to >>>> sssd.conf.bkp. However, any IPA domains from pre-installation >>>> sssd.conf should have been merged during the installation. >>>> >>>> - If sssd.conf did not exist before the installation, and no other >>>> domains than IPA domain exist in it, the patch makes sure that >>>> sssd.conf is moved to sssd.conf.deleted so user experiences no >>>> crash during any next installation due to its existence. >>>> >>>> https://fedorahosted.org/freeipa/ticket/2740 >>>> >>>> Tomas >>>> >>> Good job, SSSD uninstall process now looks more consistent and better >>> documented. I just found the following (mainly minor) issues. Comments in the >>> patch: >>> >>> diff --git a/ipa-client/ipa-install/ipa-client-install >>> b/ipa-client/ipa-install/ipa-client-install >>> index >>> 2e65921e8de2dfe68443f5b5875954d71dd48ed2..c5cef15e1fb3a3e1d7cfd070f4288d3839accfc8 >>> >>> 100755 >>> --- a/ipa-client/ipa-install/ipa-client-install >>> +++ b/ipa-client/ipa-install/ipa-client-install >>> @@ -183,6 +183,36 @@ def nssldap_exists(): >>> >>> return (retval, files_found) >>> >>> +# helper function for uninstall >>> +# deletes IPA domain from sssd.conf >>> +def delete_IPA_domain(): >>> >>> Function names should be lowercase -> delete_ipa_domain >>> >>> + sssd = ipaservices.service('sssd') >>> + try: >>> + sssdconfig = SSSDConfig.SSSDConfig() >>> + sssdconfig.import_config() >>> + domains = sssdconfig.list_active_domains() >>> + >>> + IPA_domain_name = None >>> >>> Variables should be lowercase -> ipa_domain_name >>> >>> + >>> + for name in domains: >>> + domain = sssdconfig.get_domain(name) >>> + try: >>> + provider = domain.get_option('id_provider') >>> + if provider == "ipa": >>> + IPA_domain_name = name >>> + break >>> + except SSSDConfig.NoOptionError: >>> + continue >>> + >>> + if IPA_domain_name != None: >>> >>> Do not use != with None, True, False - use "is not None". >>> >>> + sssdconfig.delete_domain(IPA_domain_name) >>> + sssdconfig.write() >>> + else: >>> + root_logger.warning("IPA domain could not be found in " + >>> + "sssd.conf and therefore not deleted") >>> + except IOError: >>> + root_logger.warning("IPA domain could not be deleted. No access to the >>> sssd.conf file.") >>> >>> There should be full path to sssd.conf in this error message. It is very useful >>> sometimes. >>> >>> + >>> def uninstall(options, env): >>> >>> if not fstore.has_files(): >>> @@ -212,7 +242,12 @@ def uninstall(options, env): >>> sssdconfig = SSSDConfig.SSSDConfig() >>> sssdconfig.import_config() >>> domains = sssdconfig.list_active_domains() >>> - if len(domains) > 1: >>> + all_domains = sssdconfig.list_domains() >>> + >>> + # we consider all the domains, because handling sssd.conf >>> + # during uninstall is dependant on was_sssd_configured flag >>> + # so the user does not lose info about inactive domains >>> + if len(all_domains) > 1: >>> # There was more than IPA domain configured >>> was_sssd_configured = True >>> for name in domains: >>> @@ -349,6 +384,62 @@ def uninstall(options, env): >>> "Failed to remove krb5/LDAP configuration: %s", str(e)) >>> return CLIENT_INSTALL_ERROR >>> >>> + # Next if-elif-elif construction deals with sssd.conf file. >>> + # Old pre-IPA domains are preserved due merging the old sssd.conf >>> + # during the installation of ipa-client but any new domains are >>> + # only present in sssd.conf now, so we don't want to delete them >>> + # by rewriting sssd.conf file. IPA domain is removed gracefully. >>> + >>> + # SSSD was installed before our installation and other non-IPA domains >>> + # found, restore backed up sssd.conf to sssd.conf.bkp and remove IPA >>> + # domain from the current sssd.conf >>> + if was_sssd_installed and was_sssd_configured: >>> + root_logger.info( >>> + "The original configuration of SSSD included other domains than " + >>> + "the IPA-based one.") >>> + >>> + delete_IPA_domain() >>> + >>> >>> + try: >>> + os.rename("/etc/sssd/sssd.conf","/etc/sssd/sssd.conf.current") >>> + if fstore.restore_file("/etc/sssd/sssd.conf"): >>> + os.rename("/etc/sssd/sssd.conf","/etc/sssd/sssd.conf.bkp") >>> + os.rename("/etc/sssd/sssd.conf.current","/etc/sssd/sssd.conf") >>> >>> I don't like this very much. I would rather not mess with /etc/sssd/sssd.conf, >>> it may surprise other tools/user and we could also end with no >>> /etc/sssd/sssd.conf at all if OSError is raised in second or third step. >>> >>> I would rather enhance fstore and implement function like "restore_file_to" >>> which would accept a second parameter with a custom path, like >>> "/etc/sssd/sssd.conf.bkp". >>> >>> + except OSError: >>> + pass >>> >>> The exception should be at least logged to debug log. >>> >>> + >>> + root_logger.info("Original pre-IPA SSSD configuration file was " + >>> + "restored to /etc/sssd/sssd.conf.bkp.") >>> + root_logger.info("IPA domain removed from current one, " + >>> + "restarting SSSD service") >>> + sssd = ipaservices.service('sssd') >>> + try: >>> + sssd.restart() >>> + except CalledProcessError: >>> + root_logger.warning("SSSD service restart was unsuccessful.") >>> + >>> + # SSSD was not installed before our installation, but other domains found, >>> + # delete IPA domain, but leave other domains intact >>> + elif not was_sssd_installed and was_sssd_configured: >>> + delete_IPA_domain() >>> + root_logger.info("Other domains than IPA domain found, " + >>> + "IPA domain was removed from sssd.conf.") >>> + try: >>> + sssd.restart() >>> + except CalledProcessError: >>> + root_logger.warning("SSSD service restart was unsuccessful.") >>> + >>> + # SSSD was not installed before our installation, and no other domains >>> + # than IPA are configured in sssd.conf - make sure config file is removed >>> + elif not was_sssd_installed and not was_sssd_configured: >>> + try: >>> + os.rename("/etc/sssd/sssd.conf","/etc/sssd/sssd.conf.deleted") >>> + except OSError: >>> + pass >>> >>> >>> Error should be logged to debug log. >>> >>> + >>> + root_logger.info("Redundant SSSD configuration file " + >>> + "/etc/sssd/sssd.conf was moved to /etc/sssd.conf.deleted") >>> + >>> if fstore.has_files(): >>> root_logger.info("Restoring client configuration files") >>> fstore.restore_all_files() >>> @@ -428,20 +519,6 @@ def uninstall(options, env): >>> if was_sshd_configured and ipaservices.knownservices.sshd.is_running(): >>> ipaservices.knownservices.sshd.restart() >>> >>> - if was_sssd_installed and was_sssd_configured: >>> - # SSSD was installed before our installation, config now is restored, >>> restart it >>> - root_logger.info( >>> - "The original configuration of SSSD included other domains than " + >>> - "the IPA-based one.") >>> - root_logger.info( >>> - "Original configuration file was restored, restarting SSSD " + >>> - "service.") >>> - sssd = ipaservices.service('sssd') >>> - try: >>> - sssd.restart() >>> - except CalledProcessError: >>> - root_logger.warning("SSSD service restart was unsuccessful.") >>> - >>> if not options.unattended: >>> root_logger.info( >>> "The original nsswitch.conf configuration has been restored.") >> Final (hopefully) version attached. >> >> Tomas > Well... almost the final version :-) > > 1) Wrong path to sssd.conf.deleted: > > #Unenrolling client from IPA server > Removing Kerberos service principals from /etc/krb5.keytab > Disabling client Kerberos and LDAP configurations > Redundant SSSD configuration file /etc/sssd/sssd.conf was moved to > /etc/sssd.conf.deleted <<<<<<<< > Restoring client configuration files > Client uninstall complete. > ipa-client-install --uninstall --unattended > > > 2) Uninstall crashes in the second case: > # ipa-client-install --uninstall --unattended > Unenrolling client from IPA server > Removing Kerberos service principals from /etc/krb5.keytab > Disabling client Kerberos and LDAP configurations > Other domains than IPA domain found, IPA domain was removed from sssd.conf. > Traceback (most recent call last): > File "/sbin/ipa-client-install", line 1913, in > sys.exit(main()) > File "/sbin/ipa-client-install", line 1891, in main > return uninstall(options, env) > File "/sbin/ipa-client-install", line 432, in uninstall > sssd.restart() > UnboundLocalError: local variable 'sssd' referenced before assignment > > > Also, full path to sssd.conf in the info message would be fine. > > > Otherwise, the patch seems to work fine. > > Martin All issues corrected. Tomas -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-tbabej-0006-4-Improves-sssd.conf-handling-during-ipa-client-uninst.patch Type: text/x-patch Size: 9110 bytes Desc: not available URL: From pspacek at redhat.com Thu Sep 20 14:36:07 2012 From: pspacek at redhat.com (Petr Spacek) Date: Thu, 20 Sep 2012 16:36:07 +0200 Subject: [Freeipa-devel] [PATCH 0065] Bump version in .spec file to 2.0 In-Reply-To: <20120920141856.GA23503@redhat.com> References: <505B2549.9080808@redhat.com> <20120920141856.GA23503@redhat.com> Message-ID: <505B29D7.7060308@redhat.com> On 09/20/2012 04:18 PM, Adam Tkac wrote: > On Thu, Sep 20, 2012 at 04:16:41PM +0200, Petr Spacek wrote: >> Hello, >> >> this patch bumps version in .spec file to 2.0. > > Ack Pushed to master: https://fedorahosted.org/bind-dyndb-ldap/changeset/bd1e312c74921f2572cad0a6ba7db7d25196d758 -- Petr^2 Spacek From mkosek at redhat.com Thu Sep 20 16:34:18 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 20 Sep 2012 18:34:18 +0200 Subject: [Freeipa-devel] [PATCH 0006] Improves sssd.conf handling during ipa-client uninstall In-Reply-To: <505B2626.4000801@redhat.com> References: <717623750.1969105.1345208663118.JavaMail.root@redhat.com> <503B5D09.7040401@redhat.com> <503B6EC8.3060508@redhat.com> <20120827133727.GG8612@zeppelin.brq.redhat.com> <503B8A65.30100@redhat.com> <503E1114.3080309@redhat.com> <5050AA68.2010300@redhat.com> <50583D07.6040001@redhat.com> <505B0F3B.3010704@redhat.com> <505B2626.4000801@redhat.com> Message-ID: <1348158858.2043.3.camel@priserak> On Thu, 2012-09-20 at 16:20 +0200, Tomas Babej wrote: > On 09/20/2012 02:42 PM, Martin Kosek wrote: > > On 09/18/2012 11:21 AM, Tomas Babej wrote: > >> On 09/12/2012 05:29 PM, Martin Kosek wrote: > >>> On 08/29/2012 02:54 PM, Tomas Babej wrote: > >>>> On 08/27/2012 04:55 PM, Martin Kosek wrote: > >>>>> On 08/27/2012 03:37 PM, Jakub Hrozek wrote: > >>>>>> On Mon, Aug 27, 2012 at 02:57:44PM +0200, Martin Kosek wrote: > >>>>>>> I think that the right behavior of SSSD conf uninstall should be the > >>>>>>> following: > >>>>>>> > >>>>>>> * sssd.conf existed before IPA install + non-IPA domains in sssd.conf found: > >>>>>>> - move backed conf up sssd.conf.bkp (and inform the user) > >>>>>>> - use SSSDConfig delete_domain function to remove ipa domain from > >>>>>>> sssd.conf > >>>>>>> - restart sssd afterwards > >>>>>> I'm confused here, which of the files is the original > >>>>>> pre-ipa-client-install file? > >>>>> This is the "backed up sssd.conf". I thought that it may be useful for user to > >>>>> still have an access to it after uninstall. > >>>>> > >>>>>> How does the non-ipa domain end up in the sssd.conf file? Does it have > >>>>>> to be configured manually or does ipa-client-install merge the list of > >>>>>> domains on installation? > >>>>> ipa-client-install merge the list of the domains. It overrides the old > >>>>> sssd.conf only when we cannot parse the sssd.conf and --preserve-sssd option > >>>>> was not set. > >>>>> > >>>>> Martin > >>>> Hi, > >>>> > >>>> The sssd.conf file is no longer left behind in case sssd was not > >>>> configured before the installation. However, the patch goes behind > >>>> the scope of this ticked and improves the handling of sssd.conf > >>>> during the ipa-client-install --uninstall in general. > >>>> > >>>> The current behaviour (well documented in source code) is as follows: > >>>> - In general, the IPA domain is simply removed from the sssd.conf > >>>> file, instead of sssd.conf being rewritten from the backup. This > >>>> preserves any domains added after installation. > >>>> > >>>> - If sssd.conf existed before the installation, it is restored to > >>>> sssd.conf.bkp. However, any IPA domains from pre-installation > >>>> sssd.conf should have been merged during the installation. > >>>> > >>>> - If sssd.conf did not exist before the installation, and no other > >>>> domains than IPA domain exist in it, the patch makes sure that > >>>> sssd.conf is moved to sssd.conf.deleted so user experiences no > >>>> crash during any next installation due to its existence. > >>>> > >>>> https://fedorahosted.org/freeipa/ticket/2740 > >>>> > >>>> Tomas > >>>> > >>> Good job, SSSD uninstall process now looks more consistent and better > >>> documented. I just found the following (mainly minor) issues. Comments in the > >>> patch: > >>> > >>> diff --git a/ipa-client/ipa-install/ipa-client-install > >>> b/ipa-client/ipa-install/ipa-client-install > >>> index > >>> 2e65921e8de2dfe68443f5b5875954d71dd48ed2..c5cef15e1fb3a3e1d7cfd070f4288d3839accfc8 > >>> > >>> 100755 > >>> --- a/ipa-client/ipa-install/ipa-client-install > >>> +++ b/ipa-client/ipa-install/ipa-client-install > >>> @@ -183,6 +183,36 @@ def nssldap_exists(): > >>> > >>> return (retval, files_found) > >>> > >>> +# helper function for uninstall > >>> +# deletes IPA domain from sssd.conf > >>> +def delete_IPA_domain(): > >>> > >>> Function names should be lowercase -> delete_ipa_domain > >>> > >>> + sssd = ipaservices.service('sssd') > >>> + try: > >>> + sssdconfig = SSSDConfig.SSSDConfig() > >>> + sssdconfig.import_config() > >>> + domains = sssdconfig.list_active_domains() > >>> + > >>> + IPA_domain_name = None > >>> > >>> Variables should be lowercase -> ipa_domain_name > >>> > >>> + > >>> + for name in domains: > >>> + domain = sssdconfig.get_domain(name) > >>> + try: > >>> + provider = domain.get_option('id_provider') > >>> + if provider == "ipa": > >>> + IPA_domain_name = name > >>> + break > >>> + except SSSDConfig.NoOptionError: > >>> + continue > >>> + > >>> + if IPA_domain_name != None: > >>> > >>> Do not use != with None, True, False - use "is not None". > >>> > >>> + sssdconfig.delete_domain(IPA_domain_name) > >>> + sssdconfig.write() > >>> + else: > >>> + root_logger.warning("IPA domain could not be found in " + > >>> + "sssd.conf and therefore not deleted") > >>> + except IOError: > >>> + root_logger.warning("IPA domain could not be deleted. No access to the > >>> sssd.conf file.") > >>> > >>> There should be full path to sssd.conf in this error message. It is very useful > >>> sometimes. > >>> > >>> + > >>> def uninstall(options, env): > >>> > >>> if not fstore.has_files(): > >>> @@ -212,7 +242,12 @@ def uninstall(options, env): > >>> sssdconfig = SSSDConfig.SSSDConfig() > >>> sssdconfig.import_config() > >>> domains = sssdconfig.list_active_domains() > >>> - if len(domains) > 1: > >>> + all_domains = sssdconfig.list_domains() > >>> + > >>> + # we consider all the domains, because handling sssd.conf > >>> + # during uninstall is dependant on was_sssd_configured flag > >>> + # so the user does not lose info about inactive domains > >>> + if len(all_domains) > 1: > >>> # There was more than IPA domain configured > >>> was_sssd_configured = True > >>> for name in domains: > >>> @@ -349,6 +384,62 @@ def uninstall(options, env): > >>> "Failed to remove krb5/LDAP configuration: %s", str(e)) > >>> return CLIENT_INSTALL_ERROR > >>> > >>> + # Next if-elif-elif construction deals with sssd.conf file. > >>> + # Old pre-IPA domains are preserved due merging the old sssd.conf > >>> + # during the installation of ipa-client but any new domains are > >>> + # only present in sssd.conf now, so we don't want to delete them > >>> + # by rewriting sssd.conf file. IPA domain is removed gracefully. > >>> + > >>> + # SSSD was installed before our installation and other non-IPA domains > >>> + # found, restore backed up sssd.conf to sssd.conf.bkp and remove IPA > >>> + # domain from the current sssd.conf > >>> + if was_sssd_installed and was_sssd_configured: > >>> + root_logger.info( > >>> + "The original configuration of SSSD included other domains than " + > >>> + "the IPA-based one.") > >>> + > >>> + delete_IPA_domain() > >>> + > >>> > >>> + try: > >>> + os.rename("/etc/sssd/sssd.conf","/etc/sssd/sssd.conf.current") > >>> + if fstore.restore_file("/etc/sssd/sssd.conf"): > >>> + os.rename("/etc/sssd/sssd.conf","/etc/sssd/sssd.conf.bkp") > >>> + os.rename("/etc/sssd/sssd.conf.current","/etc/sssd/sssd.conf") > >>> > >>> I don't like this very much. I would rather not mess with /etc/sssd/sssd.conf, > >>> it may surprise other tools/user and we could also end with no > >>> /etc/sssd/sssd.conf at all if OSError is raised in second or third step. > >>> > >>> I would rather enhance fstore and implement function like "restore_file_to" > >>> which would accept a second parameter with a custom path, like > >>> "/etc/sssd/sssd.conf.bkp". > >>> > >>> + except OSError: > >>> + pass > >>> > >>> The exception should be at least logged to debug log. > >>> > >>> + > >>> + root_logger.info("Original pre-IPA SSSD configuration file was " + > >>> + "restored to /etc/sssd/sssd.conf.bkp.") > >>> + root_logger.info("IPA domain removed from current one, " + > >>> + "restarting SSSD service") > >>> + sssd = ipaservices.service('sssd') > >>> + try: > >>> + sssd.restart() > >>> + except CalledProcessError: > >>> + root_logger.warning("SSSD service restart was unsuccessful.") > >>> + > >>> + # SSSD was not installed before our installation, but other domains found, > >>> + # delete IPA domain, but leave other domains intact > >>> + elif not was_sssd_installed and was_sssd_configured: > >>> + delete_IPA_domain() > >>> + root_logger.info("Other domains than IPA domain found, " + > >>> + "IPA domain was removed from sssd.conf.") > >>> + try: > >>> + sssd.restart() > >>> + except CalledProcessError: > >>> + root_logger.warning("SSSD service restart was unsuccessful.") > >>> + > >>> + # SSSD was not installed before our installation, and no other domains > >>> + # than IPA are configured in sssd.conf - make sure config file is removed > >>> + elif not was_sssd_installed and not was_sssd_configured: > >>> + try: > >>> + os.rename("/etc/sssd/sssd.conf","/etc/sssd/sssd.conf.deleted") > >>> + except OSError: > >>> + pass > >>> > >>> > >>> Error should be logged to debug log. > >>> > >>> + > >>> + root_logger.info("Redundant SSSD configuration file " + > >>> + "/etc/sssd/sssd.conf was moved to /etc/sssd.conf.deleted") > >>> + > >>> if fstore.has_files(): > >>> root_logger.info("Restoring client configuration files") > >>> fstore.restore_all_files() > >>> @@ -428,20 +519,6 @@ def uninstall(options, env): > >>> if was_sshd_configured and ipaservices.knownservices.sshd.is_running(): > >>> ipaservices.knownservices.sshd.restart() > >>> > >>> - if was_sssd_installed and was_sssd_configured: > >>> - # SSSD was installed before our installation, config now is restored, > >>> restart it > >>> - root_logger.info( > >>> - "The original configuration of SSSD included other domains than " + > >>> - "the IPA-based one.") > >>> - root_logger.info( > >>> - "Original configuration file was restored, restarting SSSD " + > >>> - "service.") > >>> - sssd = ipaservices.service('sssd') > >>> - try: > >>> - sssd.restart() > >>> - except CalledProcessError: > >>> - root_logger.warning("SSSD service restart was unsuccessful.") > >>> - > >>> if not options.unattended: > >>> root_logger.info( > >>> "The original nsswitch.conf configuration has been restored.") > >> Final (hopefully) version attached. > >> > >> Tomas > > Well... almost the final version :-) > > > > 1) Wrong path to sssd.conf.deleted: > > > > #Unenrolling client from IPA server > > Removing Kerberos service principals from /etc/krb5.keytab > > Disabling client Kerberos and LDAP configurations > > Redundant SSSD configuration file /etc/sssd/sssd.conf was moved to > > /etc/sssd.conf.deleted <<<<<<<< > > Restoring client configuration files > > Client uninstall complete. > > ipa-client-install --uninstall --unattended > > > > > > 2) Uninstall crashes in the second case: > > # ipa-client-install --uninstall --unattended > > Unenrolling client from IPA server > > Removing Kerberos service principals from /etc/krb5.keytab > > Disabling client Kerberos and LDAP configurations > > Other domains than IPA domain found, IPA domain was removed from sssd.conf. > > Traceback (most recent call last): > > File "/sbin/ipa-client-install", line 1913, in > > sys.exit(main()) > > File "/sbin/ipa-client-install", line 1891, in main > > return uninstall(options, env) > > File "/sbin/ipa-client-install", line 432, in uninstall > > sssd.restart() > > UnboundLocalError: local variable 'sssd' referenced before assignment > > > > > > Also, full path to sssd.conf in the info message would be fine. > > > > > > Otherwise, the patch seems to work fine. > > > > Martin > All issues corrected. > > Tomas Pushed to master, ipa-3-0. Martin From rcritten at redhat.com Thu Sep 20 17:38:24 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 20 Sep 2012 13:38:24 -0400 Subject: [Freeipa-devel] [PATCH] 1051 Fix CS replica management In-Reply-To: <505AF524.5050405@redhat.com> References: <5040F7DA.5020606@redhat.com> <505AF524.5050405@redhat.com> Message-ID: <505B5490.80303@redhat.com> Jan Cholasta wrote: > Hi, > > Dne 31.8.2012 19:43, Rob Crittenden napsal(a): >> The naming in CS replication agreements is different from IPA >> agreements, we have to live with what the create. The master side should >> be on the local side, replica1, not the remote. This required reversing >> a few master variables. >> >> Pass in the force flag to del_link. >> >> Do a better job of finding the agreements on each side. >> >> This should be ipa-csreplica-manage more in line with ipa-replica-manage. >> >> rob >> > > Rob, can you please rebase the patch on top of current master? There > were some dogtag 10 related changes to ipa-csreplica-manage since you > posted the patch. > > Honza > I re-tested after the merge and found some problems with my initial approach. The problem stems from the naming convention that dogtag uses when creating the initial agreements. It is hard to predict how things were set up later so rather than trying to reconstruct the DN we search for it and pass it when deleting agreements. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-rcrit-1051-2-csmanage.patch Type: text/x-diff Size: 9696 bytes Desc: not available URL: From rcritten at redhat.com Thu Sep 20 18:04:45 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 20 Sep 2012 14:04:45 -0400 Subject: [Freeipa-devel] [PATCH] Set master_kdc and dns_lookup_kdc to true In-Reply-To: <20120917102303.GF28400@localhost.localdomain> References: <20120907120201.GC30746@localhost.localdomain> <5053A843.7070005@redhat.com> <20120915200210.GE28400@localhost.localdomain> <1347747296.19861.78.camel@willson.li.ssimo.org> <20120917102303.GF28400@localhost.localdomain> Message-ID: <505B5ABD.5030607@redhat.com> Sumit Bose wrote: > On Sat, Sep 15, 2012 at 06:14:56PM -0400, Simo Sorce wrote: >> On Sat, 2012-09-15 at 22:02 +0200, Sumit Bose wrote: >>> On Fri, Sep 14, 2012 at 05:57:23PM -0400, Rob Crittenden wrote: >>>> Sumit Bose wrote: >>>>> Hi, >>>>> >>>>> those two patches should fix >>>>> https://fedorahosted.org/freeipa/ticket/2515 . The first makes the >>>>> needed change for fresh installations. The second adds the changes >>>>> during ipa-adtrust-install if needed. I prefer to do the changes here >>>>> instead of during updates, because during updates it is not easy to see >>>>> that the Kerberos configuration was changes. >>>>> >>>> >>>> I guess it is good form to update the RHEL 4 client installer but >>>> will anyone test it? >>> >>> I think it would be confusion if the RHEL4 client installer has >>> different information than the default one. >>> >>>> >>>> Is master_kdc supported in the MIT kfw version (krb5.ini)? >>> >>> For me it looks that the parse is build from the same sources. >>> >>>> >>>> This suffers from the problem Simo envisioned with ticket 931. If >>>> the /etc/hosts entry is removed then DNS will not start. We add an >>>> entry during installation, so this may be less of an issue. >>> >>> If the /etc/hosts entry is removed DNS will not start in either case. >>> >>> I think the solution to #931 is setting the master_kdc option. You can >>> easily reproduce startup problems if you set 'dns_lookup_kdc = true', >>> stop sssd and try to restart named. This will run into a timeout and >>> bind will not start. The reason is that besides a KDC the Kerberos >>> client libraries also try to look up the master KDC (but it seems to be >>> ok if the lookup finally fails). If sssd is running the locator plugin >>> will return the current KDC as master. If it is not running, as in the >>> test described above, /etc/krb5.conf is used next. If it does not have a >>> master_kdc entry and 'dns_lookup_kdc = false' there is no other source >>> for the master KDC and the client libraries continue with normal >>> processing. If master_kdc is not set but 'dns_lookup_kdc = true' then a >>> DNS lookup is tried, which will run into a timeout since the DNS server >>> is not started yet. But if master_kdc is set in krb5.conf the client >>> libraries will just use this value and will not try any DNS lookup, >>> independently of the setting of dns_lookup_kdc. >>> >>> As a side note. Since we run named as user named I wonder if it would be >>> possible to use SASL EXTERNAL auth instead of GSSAPI to bind to the LDAP >>> server. If this would work safe and secure there would be no >>> dependencies to the KDC during the startup of bind? >> >> The reason why we use gssapi is so that all operations performed by bind >> happen as the DNS/fqdn user, and we can use ACIs targeted at the bind >> process. In order to use SASL EXTERNAL we would need the bind process to >> change euid to an unprivileged user that we then need to map to some >> specific user. > > As said above named is already run as the unprivileged user named. > >> >> In general krb5kdc should always start before named, and should not >> depend on DNS resolution. If krb5kdc is started first bind should have >> no issues. The only proble is if gssapi libraries try to use DNS >> resolution, but we should have that solved by using the krb locator >> plugin. > > yes, and even if the locator plugin isn't available setting master_kdc > will make sure we never fall back to DNS for the local realm. > > Just to make sure, I do not want to say that the authentication type > used by named must be changes to solve potential issues. Setting > master_kdc will solve them. Ok, I'm satisfied. ACK, pushed both to master and ipa-3-0. rob From rcritten at redhat.com Thu Sep 20 21:58:37 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Thu, 20 Sep 2012 17:58:37 -0400 Subject: [Freeipa-devel] [PATCH] 1055 update audit cert renewal time In-Reply-To: <5050EFA5.7050804@redhat.com> References: <5050EFA5.7050804@redhat.com> Message-ID: <505B918D.9070302@redhat.com> Rob Crittenden wrote: > The CA audit certificate is initially valid for two years but its > profile has it renewing at six months. This bumps the value up to two > years to match the other certificates. > > This relies on Petr's and Ade's dogtag 10 patches. Updated patch. The value of policyset.caLogSigningSet.2.constraint.params.range needs to be bumped to 720 as well. rob From yzhang at redhat.com Thu Sep 20 22:37:32 2012 From: yzhang at redhat.com (yi zhang) Date: Thu, 20 Sep 2012 15:37:32 -0700 Subject: [Freeipa-devel] [PATCH] 1055 update audit cert renewal time In-Reply-To: <505B918D.9070302@redhat.com> References: <5050EFA5.7050804@redhat.com> <505B918D.9070302@redhat.com> Message-ID: <505B9AAC.8080603@redhat.com> On 09/20/2012 02:58 PM, Rob Crittenden wrote: > Updated patch. The value of > policyset.caLogSigningSet.2.constraint.params.range needs to be bumped > to 720 as well. I keep doing my test and let everyone know the test result. Yi -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | Yi Zhang | | QA @ Mountain View, California | | Cell: 408-509-6375 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -------------- next part -------------- A non-text attachment was scrubbed... Name: yzhang.vcf Type: text/x-vcard Size: 134 bytes Desc: not available URL: From pspacek at redhat.com Fri Sep 21 08:13:37 2012 From: pspacek at redhat.com (Petr Spacek) Date: Fri, 21 Sep 2012 10:13:37 +0200 Subject: [Freeipa-devel] [PATCH 0066] Log errors from dns_name_concatenate() in zone_refresh() properly Message-ID: <505C21B1.70302@redhat.com> Hello, Log errors from dns_name_concatenate() in zone_refresh() properly. -- Petr^2 Spacek -------------- next part -------------- A non-text attachment was scrubbed... Name: bind-dyndb-ldap-pspacek-0066-Log-errors-from-dns_name_concatenate-in-zone_refresh.patch Type: text/x-patch Size: 1088 bytes Desc: not available URL: From atkac at redhat.com Fri Sep 21 09:07:52 2012 From: atkac at redhat.com (Adam Tkac) Date: Fri, 21 Sep 2012 11:07:52 +0200 Subject: [Freeipa-devel] [PATCH 0066] Log errors from dns_name_concatenate() in zone_refresh() properly In-Reply-To: <505C21B1.70302@redhat.com> References: <505C21B1.70302@redhat.com> Message-ID: <20120921090751.GA8188@redhat.com> On Fri, Sep 21, 2012 at 10:13:37AM +0200, Petr Spacek wrote: > Hello, > > Log errors from dns_name_concatenate() in zone_refresh() properly. Ack. > From b8abc442b0d483366f002a8364a333577042d652 Mon Sep 17 00:00:00 2001 > From: Petr Spacek > Date: Fri, 21 Sep 2012 10:10:55 +0200 > Subject: [PATCH] Log errors from dns_name_concatenate() in zone_refresh() > properly. > > Signed-off-by: Petr Spacek > --- > src/ldap_helper.c | 8 +++++--- > 1 file changed, 5 insertions(+), 3 deletions(-) > > diff --git a/src/ldap_helper.c b/src/ldap_helper.c > index 2245cb982f26eab165a327b4ad72046f9eb4024e..3301f8d872239dcc48506d1f935dacad7bf5990f 100644 > --- a/src/ldap_helper.c > +++ b/src/ldap_helper.c > @@ -1312,9 +1312,11 @@ refresh_zones_from_ldap(ldap_instance_t *ldap_inst, isc_boolean_t delete_only) > goto next; > } > > - if (dns_name_concatenate(&fname, &forig, &aname, aname.buffer) > - != ISC_R_SUCCESS) { > - log_error_r("unable to concatenate DNS names during zone_refresh"); > + result = dns_name_concatenate(&fname, &forig, &aname, > + aname.buffer); > + if (result != ISC_R_SUCCESS) { > + log_error_r("unable to concatenate DNS names" > + "during zone_refresh"); > goto next; > } > > -- > 1.7.11.4 > -- Adam Tkac, Red Hat, Inc. From pspacek at redhat.com Fri Sep 21 10:27:27 2012 From: pspacek at redhat.com (Petr Spacek) Date: Fri, 21 Sep 2012 12:27:27 +0200 Subject: [Freeipa-devel] [PATCH 0066] Log errors from dns_name_concatenate() in zone_refresh() properly In-Reply-To: <20120921090751.GA8188@redhat.com> References: <505C21B1.70302@redhat.com> <20120921090751.GA8188@redhat.com> Message-ID: <505C410F.1010808@redhat.com> On 09/21/2012 11:07 AM, Adam Tkac wrote: > On Fri, Sep 21, 2012 at 10:13:37AM +0200, Petr Spacek wrote: >> Hello, >> >> Log errors from dns_name_concatenate() in zone_refresh() properly. > > Ack. Pushed to master: https://fedorahosted.org/bind-dyndb-ldap/changeset/7710d89b58d3c7fe715c9177bb73d695119245ae Petr^2 Spacek From rcritten at redhat.com Fri Sep 21 13:21:43 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Fri, 21 Sep 2012 09:21:43 -0400 Subject: [Freeipa-devel] [PATCH] 0077 Check direct/reverse hostname/address resolution in ipa-replica-install In-Reply-To: <505AD4BB.9040509@redhat.com> References: <5046009E.2020306@redhat.com> <50463DE6.9070300@redhat.com> <5046FF0E.1000504@redhat.com> <504FA7A3.5080201@redhat.com> <5050B1ED.70307@redhat.com> <50524387.4020800@redhat.com> <5052D2CC.2000405@redhat.com> <50570D68.9040501@redhat.com> <505767AA.5080607@redhat.com> <50585832.8060005@redhat.com> <5059DD16.9010202@redhat.com> <5059E87B.2070500@redhat.com> <505A1319.8010803@redhat.com> <505AD4BB.9040509@redhat.com> Message-ID: <505C69E7.9050808@redhat.com> Petr Viktorin wrote: > On 09/19/2012 08:46 PM, Rob Crittenden wrote: >> Petr Viktorin wrote: >>> On 09/19/2012 04:56 PM, Rob Crittenden wrote: >>>> Petr Viktorin wrote: >>>>> On 09/17/2012 08:10 PM, Rob Crittenden wrote: >>>>>> Petr Viktorin wrote: >>>>>>> On 09/14/2012 08:46 AM, Martin Kosek wrote: >>>>>>>> On 09/13/2012 10:35 PM, Rob Crittenden wrote: >>>>>>>>> Petr Viktorin wrote: >>>>>>>>>> On 09/11/2012 11:05 PM, Rob Crittenden wrote: >>>>>>>>>>> Petr Viktorin wrote: >>>>>>>>>>>> On 09/04/2012 07:44 PM, Rob Crittenden wrote: >>>>>>>>>>>>> Petr Viktorin wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://fedorahosted.org/freeipa/ticket/2845 >>>>>>>>>>>>> >>>>>>>>>>>>> Shouldn't this also call verify_fqdn() on the local hostname >>>>>>>>>>>>> and >>>>>>>>>>>>> not >>>>>>>>>>>>> just the master? I think this would eventually fail in the >>>>>>>>>>>>> conncheck >>>>>>>>>>>>> but >>>>>>>>>>>>> what if that was skipped? >>>>>>>>>>>>> >>>>>>>>>>>>> rob >>>>>>>>>>>> >>>>>>>>>>>> A few lines above there is a call to get_host_name, which will >>>>>>>>>>>> call >>>>>>>>>>>> verify_fqdn. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I double-checked this, it fails in conncheck. Here are my steps: >>>>>>>>>>> >>>>>>>>>>> # ipa-server-install --setup-dns >>>>>>>>>>> # ipa-replica-prepare replica.example.com >>>>>>>>>>> --ip-address=192.168.100.2 >>>>>>>>>>> # ipa host-del replica.example.com >>>>>>>>>>> >>>>>>>>>>> On replica, set DNS to IPA master, with hostname in /etc/hosts. >>>>>>>>>>> >>>>>>>>>>> # ipa-replica-install ... >>>>>>>>>>> >>>>>>>>>>> The verify_fqdn() passes because the resolver uses /etc/hosts. >>>>>>>>>>> >>>>>>>>>>> The conncheck fails: >>>>>>>>>>> >>>>>>>>>>> Execute check on remote master >>>>>>>>>>> Check connection from master to remote replica >>>>>>>>>>> 'replica.example.com': >>>>>>>>>>> >>>>>>>>>>> Remote master check failed with following error message(s): >>>>>>>>>>> Could not chdir to home directory /home/admin: No such file or >>>>>>>>>>> directory >>>>>>>>>>> Port check failed! Unable to resolve host name >>>>>>>>>>> 'replica.example.com' >>>>>>>>>>> >>>>>>>>>>> Connection check failed! >>>>>>>>>>> Please fix your network settings according to error messages >>>>>>>>>>> above. >>>>>>>>>>> If the check results are not valid it can be skipped with >>>>>>>>>>> --skip-conncheck parameter. >>>>>>>>>>> >>>>>>>>>>> The DNS test happens much further after this, and I get why, I >>>>>>>>>>> just >>>>>>>>>>> don't see how useful it is unless the --skip-conncheck is used. >>>>>>>>>> >>>>>>>>>> For the record, it's because we need to check if the host has DNS >>>>>>>>>> installed. We need a LDAP connection to check this. >>>>>>>>>> >>>>>>>>>>> ipa-replica-install ~rcrit/replica-info-replica.example.com.gpg >>>>>>>>>>> --skip-conncheck >>>>>>>>>>> Directory Manager (existing master) password: >>>>>>>>>>> >>>>>>>>>>> ipa : ERROR Could not resolve hostname >>>>>>>>>>> replica.example.com >>>>>>>>>>> using DNS. Clients may not function properly. Please check your >>>>>>>>>>> DNS >>>>>>>>>>> setup. (Note that this check queries IPA DNS directly and >>>>>>>>>>> ignores >>>>>>>>>>> /etc/hosts.) >>>>>>>>>>> Continue? [no]: >>>>>>>>>>> >>>>>>>>>>> So I guess, what are the intentions here? It is certainly better >>>>>>>>>>> than >>>>>>>>>>> before. >>>>>>>>>>> >>>>>>>>>>> rob >>>>>>>>>> >>>>>>>>>> If the replica is in the master's /etc/hosts, but not in DNS, the >>>>>>>>>> conncheck will succeed. This check explicitly queries IPA records >>>>>>>>>> only >>>>>>>>>> and ignores /etc/hosts so it'll notice this case and warn. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Ok, like I said, this is better than we have. Just one nit then >>>>>>>>> you >>>>>>>>> get an ack: >>>>>>>>> >>>>>>>>> + # If remote host has DNS, check forward/reverse >>>>>>>>> resolution >>>>>>>>> + try: >>>>>>>>> + entry = conn.find_entries(u'cn=dns', >>>>>>>>> base_dn=DN(api.env.basedn)) >>>>>>>>> + except errors.NotFound: >>>>>>>>> >>>>>>>>> u'cn=dns' should be str(constants.container_dns). >>>>>>>>> >>>>>>>>> rob >>>>>>>> >>>>>>>> This is a search filter, Petr could use the one I already have in >>>>>>>> "dns.py::get_dns_masters()" function: >>>>>>>> '(&(objectClass=ipaConfigObject)(cn=DNS))' >>>>>>>> >>>>>>>> For performance sake, I would also not search in the entire tree, >>>>>>>> but >>>>>>>> limit the >>>>>>>> search only to: >>>>>>>> >>>>>>>> DN(('cn', 'masters'), ('cn', 'ipa'), ('cn', 'etc'), api.env.basedn) >>>>>>>> >>>>>>>> Martin >>>>>>>> >>>>>>> >>>>>>> Attaching updated patch with Martin's suggestions. >>>>>> >>>>>> I think what Martin had in mind was: >>>>>> >>>>>> if api.Object.dnsrecord.get_dns_masters(): >>>>>> ... >>>>>> >>>>> >>>>> I didn't want to do this because api.Object.* use our global ldap2 >>>>> Backend, which is hardwired to query localhost. >>>>> I see now that I can hack around this, and we already do this in >>>>> ipa-replica-install. >>>>> I've extracted the hack and reused it to get the DNS masters. >>>>> >>>>> >>>> >>>> I can't say I'm crazy about the method name you've chosen... >>>> >>>> rob >>> >>> I intended the name as a warning to not use it unless necessary. >>> >>> Changed to temporary_ldap2_connection. >>> >> >> I found a dangling reference to replman. I removed this and installation >> seemed to work ok. >> >> --- install/tools/ipa-replica-install 2012-09-19 14:01:16.169053047 >> -0400 >> +++ /usr/sbin/ipa-replica-install 2012-09-19 14:43:06.684917906 >> -0400 >> @@ -564,8 +564,6 @@ >> finally: >> if conn and conn.isconnected(): >> conn.disconnect() >> - if replman and replman.conn: >> - replman.conn.unbind_s() >> >> # Configure ntpd >> if options.conf_ntp: >> > > I never had a problem with this. How are you installing the replica? > It's true that replman might be uninitialized if there's an error > (though it doesn't seem related to the issue). Attached patch > initializes it. > ACK, pushed to master and ipa-3-0 rob From pspacek at redhat.com Fri Sep 21 13:26:27 2012 From: pspacek at redhat.com (Petr Spacek) Date: Fri, 21 Sep 2012 15:26:27 +0200 Subject: [Freeipa-devel] [PATCH 0067] Fix error handling in ldap_get_zone_serial() Message-ID: <505C6B03.7020809@redhat.com> Hello, Fix error handling in ldap_get_zone_serial(). Successful call of zr_get_zone_ptr() attaches zone. Zone has to be detached regardless of dns_zone_getserial2() return code. Petr^2 Spacek -------------- next part -------------- A non-text attachment was scrubbed... Name: bind-dyndb-ldap-pspacek-0067-Fix-error-handling-in-ldap_get_zone_serial.patch Type: text/x-patch Size: 983 bytes Desc: not available URL: From pspacek at redhat.com Fri Sep 21 13:30:41 2012 From: pspacek at redhat.com (Petr Spacek) Date: Fri, 21 Sep 2012 15:30:41 +0200 Subject: [Freeipa-devel] [PATCH 0061] Add missing DNS view attach/detach to LDAP instance management code In-Reply-To: <505AFB52.8040206@redhat.com> References: <5051C581.9060209@redhat.com> <505AFB52.8040206@redhat.com> Message-ID: <505C6C01.6070400@redhat.com> On 09/20/2012 01:17 PM, Petr Spacek wrote: > On 09/13/2012 01:37 PM, Petr Spacek wrote: >> Hello, >> >> Add missing DNS view attach/detach to LDAP instance management code. >> This fixes race condition in BIND shutdown after SIGINT: >> - failing assert caused by use-after-free in dns_zt_find(): >> (((zt) != ((void *)0)) && (((const isc__magic_t *)(zt))->magic >> == ((('Z') << 24 | ('T') << 16 | ('b') << 8 | ('l'))))) >> >> Petr^2 Spacek > > This patch causes deadlock in shut-down sequence in certain situations, I'm > still searching for the root cause. > > Old comment from 2009-02-10 > /* commented out for now, cause named to hang */ > was correct, apparently ... > Apparently my statement above was wrong. Deadlock I observed was caused by incorrect zone attach/detach sequence. This problem is fixed in "[PATCH 0067] Fix error handling in ldap_get_zone_serial()". You can proceed with reviewing this patch. Petr^2 Spacek From tjaalton at ubuntu.com Fri Sep 21 14:57:15 2012 From: tjaalton at ubuntu.com (Timo Aaltonen) Date: Fri, 21 Sep 2012 17:57:15 +0300 Subject: [Freeipa-devel] [PATCH] convert the base platform modules into packages Message-ID: <505C804B.9010506@ubuntu.com> Ok, so this is the first step before we can start to rewrite bits from ipaserver/install to make them support other distros. There are no real functional changes yet. had some dependency issues installing the resulting rpm's, so didn't test the install scripts but they should work :) -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-convert-the-base-platform-modules-into-packages.patch Type: text/x-patch Size: 52908 bytes Desc: not available URL: From pviktori at redhat.com Mon Sep 24 11:03:04 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Mon, 24 Sep 2012 13:03:04 +0200 Subject: [Freeipa-devel] [PATCH] 0083 Always handle NotFound error in dnsrecord-mod Message-ID: <50603DE8.2010208@redhat.com> When there were no updated attrs when modifying a nonexistent DNS record, the error was not handled and caused an internal server error later (old_entry was used uninitialized). https://fedorahosted.org/freeipa/ticket/3055 -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0083-Always-handle-NotFound-error-in-dnsrecord-mod.patch Type: text/x-patch Size: 2174 bytes Desc: not available URL: From pspacek at redhat.com Mon Sep 24 11:22:32 2012 From: pspacek at redhat.com (Petr Spacek) Date: Mon, 24 Sep 2012 13:22:32 +0200 Subject: [Freeipa-devel] [PATCH 0068] Fix "unable to concatenate DNS names during zone_refresh" error Message-ID: <50604278.7020705@redhat.com> Hello, Fix "unable to concatenate DNS names during zone_refresh" error. This fixes zone refresh for cases where concatenation of all zone names is longer than 255 characters. -- Petr^2 Spacek -------------- next part -------------- A non-text attachment was scrubbed... Name: bind-dyndb-ldap-pspacek-0068-Fix-unable-to-concatenate-DNS-names-during-zone_refr.patch Type: text/x-patch Size: 1529 bytes Desc: not available URL: From mkosek at redhat.com Mon Sep 24 12:00:44 2012 From: mkosek at redhat.com (Martin Kosek) Date: Mon, 24 Sep 2012 14:00:44 +0200 Subject: [Freeipa-devel] [PATCH] 0083 Always handle NotFound error in dnsrecord-mod In-Reply-To: <50603DE8.2010208@redhat.com> References: <50603DE8.2010208@redhat.com> Message-ID: <50604B6C.8010202@redhat.com> On 09/24/2012 01:03 PM, Petr Viktorin wrote: > When there were no updated attrs when modifying a nonexistent DNS record, the > error was not handled and caused an internal server error later (old_entry was > used uninitialized). > > https://fedorahosted.org/freeipa/ticket/3055 > ACK. Pushed to master, ipa-3-0. Martin From atkac at redhat.com Mon Sep 24 12:08:08 2012 From: atkac at redhat.com (Adam Tkac) Date: Mon, 24 Sep 2012 14:08:08 +0200 Subject: [Freeipa-devel] [PATCH 0062] Prevent memory read outside allocated space in str_alloc() In-Reply-To: <5052EFF0.1040805@redhat.com> References: <5052EFF0.1040805@redhat.com> Message-ID: <20120924120807.GA14825@redhat.com> On Fri, Sep 14, 2012 at 10:50:56AM +0200, Petr Spacek wrote: > Hello, > > Prevent memory read outside allocated space in str_alloc(). > > Found by Valgrind during nsupdate stress test. Ack > From c53ec9cf2cc22e29630767b6b2259d145192ff62 Mon Sep 17 00:00:00 2001 > From: Petr Spacek > Date: Fri, 14 Sep 2012 10:48:04 +0200 > Subject: [PATCH] Prevent memory read outside allocated space in str_alloc(). > > Signed-off-by: Petr Spacek > --- > src/str.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/src/str.c b/src/str.c > index 0096536de071f7d0dd2ff327e4d6ac72e81dab5f..83645365ee6eff7bda5fbeda6837f30d4dec41ae 100644 > --- a/src/str.c > +++ b/src/str.c > @@ -102,7 +102,7 @@ str_alloc(ld_string_t *str, size_t len) > return ISC_R_NOMEMORY; > > if (str->data != NULL) { > - memcpy(new_buffer, str->data, len); > + memcpy(new_buffer, str->data, str->allocated); > new_buffer[len] = '\0'; > isc_mem_put(str->mctx, str->data, str->allocated); > } else { > -- > 1.7.11.4 > -- Adam Tkac, Red Hat, Inc. From atkac at redhat.com Mon Sep 24 12:19:13 2012 From: atkac at redhat.com (Adam Tkac) Date: Mon, 24 Sep 2012 14:19:13 +0200 Subject: [Freeipa-devel] [PATCH 0054] Allow BIND to start if LDAP connection times out In-Reply-To: <503CB2F9.7080603@redhat.com> References: <503CB2F9.7080603@redhat.com> Message-ID: <20120924121913.GA15023@redhat.com> On Tue, Aug 28, 2012 at 02:00:57PM +0200, Petr Spacek wrote: > Hello, > > this patch allows BIND to start if LDAP connection times out. BIND > will reconnect in same way as after "connection refused" errors. > > The patch closes https://fedorahosted.org/bind-dyndb-ldap/ticket/84 . Ack > From eaa35060fc47c1422ca7b577fe0096aadd2f8c0a Mon Sep 17 00:00:00 2001 > From: Petr Spacek > Date: Tue, 28 Aug 2012 13:54:51 +0200 > Subject: [PATCH] Allow BIND to start if LDAP connection times out. > > https://fedorahosted.org/bind-dyndb-ldap/ticket/84 > > Signed-off-by: Petr Spacek > --- > src/ldap_helper.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/src/ldap_helper.c b/src/ldap_helper.c > index da083d2e65032e650cfbbeb863262e0141403407..d533d1985c9fb8f56f333a99208d3f60f0b4c5bf 100644 > --- a/src/ldap_helper.c > +++ b/src/ldap_helper.c > @@ -2797,7 +2797,7 @@ ldap_pool_connect(ldap_pool_t *pool, ldap_instance_t *ldap_inst) > ldap_conn = NULL; > CHECK(new_ldap_connection(pool, &ldap_conn)); > result = ldap_connect(ldap_inst, ldap_conn, ISC_FALSE); > - if (result == ISC_R_NOTCONNECTED) { > + if (result == ISC_R_NOTCONNECTED || result == ISC_R_TIMEDOUT) { > /* LDAP server is down which can happen, continue */ > result = ISC_R_SUCCESS; > } else if (result != ISC_R_SUCCESS) { > -- > 1.7.11.2 > -- Adam Tkac, Red Hat, Inc. From atkac at redhat.com Mon Sep 24 13:00:36 2012 From: atkac at redhat.com (Adam Tkac) Date: Mon, 24 Sep 2012 15:00:36 +0200 Subject: [Freeipa-devel] [PATCH 0060] Fix zone delete in ldap_zone_delete2() In-Reply-To: <5051C540.1030702@redhat.com> References: <5051C540.1030702@redhat.com> Message-ID: <20120924130035.GA15537@redhat.com> On Thu, Sep 13, 2012 at 01:36:32PM +0200, Petr Spacek wrote: > Hello, > > Fix zone delete in ldap_zone_delete2(). This fixes two race > conditions during BIND reload: > > - failing assert in destroy_ldap_connection() DESTROYLOCK: > ((pthread_mutex_destroy(&ldap_conn->lock) == 0) ? 0 : 34) == 0 > > - use-after-free in call: > ldap_cache_enabled(cache=0xdededededededede) Ack. When pushing, please consider if "race-condition" is good description. From my point of view better is "fix extraction of zone FQDN when destroing LDAP instance" or something like that. Regards, Adam > From dc017b4d7250289eb5938262dbb43632126f9329 Mon Sep 17 00:00:00 2001 > From: Petr Spacek > Date: Thu, 13 Sep 2012 13:02:19 +0200 > Subject: [PATCH] Fix zone delete in ldap_zone_delete2(). This fixes two race > conditions during BIND reload: > > - failing assert in destroy_ldap_connection() DESTROYLOCK: > ((pthread_mutex_destroy(&ldap_conn->lock) == 0) ? 0 : 34) == 0 > > - use-after-free in call: > ldap_cache_enabled(cache=0xdededededededede) > > Signed-off-by: Petr Spacek > --- > src/ldap_helper.c | 64 +++++++++++++++++++++++++++++++++++++++++-------------- > 1 file changed, 48 insertions(+), 16 deletions(-) > > diff --git a/src/ldap_helper.c b/src/ldap_helper.c > index 67a64b79159983c83cb1bfc73c4b02a9bce986a7..3b49de809738fef18cae10c38fd3d9c33eef5141 100644 > --- a/src/ldap_helper.c > +++ b/src/ldap_helper.c > @@ -517,45 +517,68 @@ destroy_ldap_instance(ldap_instance_t **ldap_instp) > ldap_instance_t *ldap_inst; > dns_rbtnodechain_t chain; > dns_rbt_t *rbt; > - isc_result_t result; > + isc_result_t result = ISC_R_SUCCESS; > + const char *db_name; > > REQUIRE(ldap_instp != NULL && *ldap_instp != NULL); > > ldap_inst = *ldap_instp; > + db_name = ldap_inst->db_name; /* points to DB instance: outside ldap_inst */ > > /* > * Unregister all zones already registered in BIND. > * > * NOTE: This should be probably done in zone_register.c > */ > - dns_rbtnodechain_init(&chain, ldap_inst->mctx); > rbt = zr_get_rbt(ldap_inst->zone_register); > > /* Potentially ISC_R_NOSPACE can occur. Destroy codepath has no way to > * return errors, so kill BIND. > * DNS_R_NAMETOOLONG should never happen, because all names were checked > * while loading. */ > - result = dns_rbtnodechain_first(&chain, rbt, NULL, NULL); > - RUNTIME_CHECK(result == ISC_R_SUCCESS || result == DNS_R_NEWORIGIN > - || result == ISC_R_NOTFOUND); > > + dns_rbtnodechain_init(&chain, ldap_inst->mctx); > while (result != ISC_R_NOMORE && result != ISC_R_NOTFOUND) { > dns_fixedname_t name; > + dns_fixedname_t origin; > + dns_fixedname_t concat; > dns_fixedname_init(&name); > - result = dns_rbtnodechain_current(&chain, NULL, > - dns_fixedname_name(&name), > - NULL); > - RUNTIME_CHECK(result == ISC_R_SUCCESS); > + dns_fixedname_init(&origin); > + dns_fixedname_init(&concat); > + > + dns_rbtnodechain_reset(&chain); > + result = dns_rbtnodechain_first(&chain, rbt, NULL, NULL); > + RUNTIME_CHECK(result == ISC_R_SUCCESS || result == DNS_R_NEWORIGIN || > + result == ISC_R_NOTFOUND); > + > + /* Search for first zone in zone register and omit auxiliary nodes. */ > + while (result != ISC_R_NOMORE && result != ISC_R_NOTFOUND) { > + dns_rbtnode_t *node = NULL; > + > + result = dns_rbtnodechain_current(&chain, dns_fixedname_name(&name), > + dns_fixedname_name(&origin), &node); > + RUNTIME_CHECK(result == ISC_R_SUCCESS); > + > + if (node->data != NULL) { /* Auxiliary nodes have data == NULL. */ > + result = dns_name_concatenate(dns_fixedname_name(&name), > + dns_fixedname_name(&origin), > + dns_fixedname_name(&concat), > + NULL); > + RUNTIME_CHECK(result == ISC_R_SUCCESS); > + break; > + } > + > + result = dns_rbtnodechain_next(&chain, NULL, NULL); > + RUNTIME_CHECK(result == ISC_R_SUCCESS || result == DNS_R_NEWORIGIN || > + result == ISC_R_NOMORE); > + } > + if (result == ISC_R_NOMORE || result == ISC_R_NOTFOUND) > + break; > > result = ldap_delete_zone2(ldap_inst, > - dns_fixedname_name(&name), > + dns_fixedname_name(&concat), > ISC_TRUE); > - RUNTIME_CHECK(result == ISC_R_SUCCESS); > - > - result = dns_rbtnodechain_next(&chain, NULL, NULL); > - RUNTIME_CHECK(result == ISC_R_SUCCESS || > - result == DNS_R_NEWORIGIN || > - result == ISC_R_NOMORE); > + RUNTIME_CHECK(result == ISC_R_SUCCESS); > } > > dns_rbtnodechain_invalidate(&chain); > @@ -606,6 +629,7 @@ destroy_ldap_instance(ldap_instance_t **ldap_instp) > isc_mem_putanddetach(&ldap_inst->mctx, ldap_inst, sizeof(ldap_instance_t)); > > *ldap_instp = NULL; > + log_debug(1, "LDAP instance '%s' destroyed", db_name); > } > > static isc_result_t > @@ -776,7 +800,10 @@ ldap_delete_zone2(ldap_instance_t *inst, dns_name_t *name, isc_boolean_t lock) > isc_boolean_t freeze = ISC_FALSE; > dns_zone_t *zone = NULL; > dns_zone_t *foundzone = NULL; > + char zone_name_char[DNS_NAME_FORMATSIZE]; > > + dns_name_format(name, zone_name_char, DNS_NAME_FORMATSIZE); > + log_debug(1, "deleting zone '%s'", zone_name_char); > if (lock) { > result = isc_task_beginexclusive(inst->task); > RUNTIME_CHECK(result == ISC_R_SUCCESS || > @@ -790,6 +817,7 @@ ldap_delete_zone2(ldap_instance_t *inst, dns_name_t *name, isc_boolean_t lock) > > result = zr_get_zone_ptr(inst->zone_register, name, &zone); > if (result == ISC_R_NOTFOUND) { > + log_debug(1, "zone '%s' not found in zone register", zone_name_char); > result = ISC_R_SUCCESS; > goto cleanup; > } else if (result != ISC_R_SUCCESS) > @@ -810,7 +838,11 @@ ldap_delete_zone2(ldap_instance_t *inst, dns_name_t *name, isc_boolean_t lock) > if (dns_zone_getdb(zone, &dbp) == ISC_R_SUCCESS) { > dns_db_detach(&dbp); /* dns_zone_getdb() attaches DB implicitly */ > dns_zone_unload(zone); > + log_debug(1, "zone '%s' unloaded", zone_name_char); > + } else { > + log_debug(1, "zone '%s' not loaded - unload skipped", zone_name_char); > } > + > CHECK(dns_zt_unmount(inst->view->zonetable, zone)); > CHECK(zr_del_zone(inst->zone_register, name)); > dns_zonemgr_releasezone(inst->zmgr, zone); > -- > 1.7.11.4 > -- Adam Tkac, Red Hat, Inc. From atkac at redhat.com Mon Sep 24 13:09:41 2012 From: atkac at redhat.com (Adam Tkac) Date: Mon, 24 Sep 2012 15:09:41 +0200 Subject: [Freeipa-devel] [PATCH 0063] Notify DNS slaves if zone serial number modification was detected. In-Reply-To: <50571DAA.5030506@redhat.com> References: <50571DAA.5030506@redhat.com> Message-ID: <20120924130939.GA15761@redhat.com> On Mon, Sep 17, 2012 at 02:55:06PM +0200, Petr Spacek wrote: > Hello, > > this patch adds missing notification to DNS slaves if zone serial > number modification was detected. Hi, please check my comment below. > From eb8d7fc0c02e03b9c7c90e497225536c449fab1c Mon Sep 17 00:00:00 2001 > From: Petr Spacek > Date: Mon, 17 Sep 2012 14:29:45 +0200 > Subject: [PATCH] Notify DNS slaves if zone serial number modification was > detected. > > Signed-off-by: Petr Spacek > --- > src/ldap_helper.c | 22 ++++++++++++++-------- > 1 file changed, 14 insertions(+), 8 deletions(-) > > diff --git a/src/ldap_helper.c b/src/ldap_helper.c > index 658b960f50b461fa602edf51e955f3bdd4769e1d..d22e8714df57edaad4cf45e6cc26ec0dbbd59108 100644 > --- a/src/ldap_helper.c > +++ b/src/ldap_helper.c > @@ -1035,9 +1035,10 @@ ldap_parse_zoneentry(ldap_entry_t *entry, ldap_instance_t *inst) > isc_task_t *task = inst->task; > isc_uint32_t ldap_serial; > isc_uint32_t zr_serial; /* SOA serial value from in-memory zone register */ > - unsigned char ldap_digest[RDLIST_DIGESTLENGTH]; > + unsigned char ldap_digest[RDLIST_DIGESTLENGTH] = {0}; > unsigned char *zr_digest = NULL; > ldapdb_rdatalist_t rdatalist; > + isc_boolean_t zone_dynamic = ISC_FALSE; > > REQUIRE(entry != NULL); > REQUIRE(inst != NULL); > @@ -1131,13 +1132,13 @@ ldap_parse_zoneentry(ldap_entry_t *entry, ldap_instance_t *inst) > && result != DNS_R_DYNAMIC && result != DNS_R_CONTINUE) > goto cleanup; > > + zone_dynamic = (result == DNS_R_DYNAMIC); > result = ISC_R_SUCCESS; > > /* initialize serial in zone register and always increment serial > * for a new zone (typically after BIND start) > * - the zone was possibly changed in meanwhile */ > if (publish) { > - memset(ldap_digest, 0, RDLIST_DIGESTLENGTH); > CHECK(ldap_get_zone_serial(inst, &name, &ldap_serial)); > CHECK(zr_set_zone_serial_digest(inst->zone_register, &name, ldap_serial, > ldap_digest)); > @@ -1154,23 +1155,28 @@ ldap_parse_zoneentry(ldap_entry_t *entry, ldap_instance_t *inst) > * 3c) The old and new serials are same: autoincrement only if something > * else was changed. > */ > + CHECK(ldap_get_zone_serial(inst, &name, &ldap_serial)); > + CHECK(zr_get_zone_serial_digest(inst->zone_register, &name, &zr_serial, > + &zr_digest)); > if (inst->serial_autoincrement) { > - CHECK(ldap_get_zone_serial(inst, &name, &ldap_serial)); > CHECK(ldapdb_rdatalist_get(inst->mctx, inst, &name, > &name, &rdatalist)); > CHECK(rdatalist_digest(inst->mctx, &rdatalist, ldap_digest)); > > - CHECK(zr_get_zone_serial_digest(inst->zone_register, &name, &zr_serial, > - &zr_digest)); > if (ldap_serial == zr_serial) { > /* serials are same - increment only if something was changed */ > if (memcmp(zr_digest, ldap_digest, RDLIST_DIGESTLENGTH) != 0) > CHECK(soa_serial_increment(inst->mctx, inst, &name)); > - } else { /* serial in LDAP was changed - update zone register */ > - CHECK(zr_set_zone_serial_digest(inst->zone_register, &name, > - ldap_serial, ldap_digest)); > } > } > + if (ldap_serial != zr_serial) { > + /* serial in LDAP was changed - update zone register */ > + CHECK(zr_set_zone_serial_digest(inst->zone_register, &name, > + ldap_serial, ldap_digest)); > + > + if (zone_dynamic) > + dns_zone_notify(zone); This if() doesn't seem fully correct to me. Although in most FreeIPA scenarios it will work, consider situation when someone disables DDNS updates for zone and modifies records in LDAP. In this case zone_dynamic is FALSE but SOA serial changes. I would recommend to simpy send notify every time when serial changes. Regards, Adam > + } > > cleanup: > if (unlock) > -- > 1.7.11.4 > -- Adam Tkac, Red Hat, Inc. From atkac at redhat.com Mon Sep 24 13:15:17 2012 From: atkac at redhat.com (Adam Tkac) Date: Mon, 24 Sep 2012 15:15:17 +0200 Subject: [Freeipa-devel] [PATCH 0064] Improve log message about improperly formated Resource Records In-Reply-To: <50573CC0.8050308@redhat.com> References: <50573CC0.8050308@redhat.com> Message-ID: <20120924131517.GB15761@redhat.com> On Mon, Sep 17, 2012 at 05:07:44PM +0200, Petr Spacek wrote: > Hello, > > this patch adds DN to log message about improperly formated Resource Records. Hi, please check my comment below, otherwise ack. Regards, Adam > From d36ae54c593c33a45ef9936720357ff7de30c8b5 Mon Sep 17 00:00:00 2001 > From: Petr Spacek > Date: Mon, 17 Sep 2012 17:01:41 +0200 > Subject: [PATCH] Improve log message about improperly formated Resource > Records. > > Signed-off-by: Petr Spacek > --- > src/ldap_helper.c | 17 ++++++++++------- > 1 file changed, 10 insertions(+), 7 deletions(-) > > diff --git a/src/ldap_helper.c b/src/ldap_helper.c > index d22e8714df57edaad4cf45e6cc26ec0dbbd59108..2245cb982f26eab165a327b4ad72046f9eb4024e 100644 > --- a/src/ldap_helper.c > +++ b/src/ldap_helper.c > @@ -1473,6 +1473,8 @@ ldap_parse_rrentry(isc_mem_t *mctx, ldap_entry_t *entry, > dns_rdata_t *rdata = NULL; > dns_rdatalist_t *rdlist = NULL; > ldap_attribute_t *attr; > + const char *dn = ""; > + const char *data = ""; > > result = add_soa_record(mctx, qresult, origin, entry, > rdatalist, fake_mname); > @@ -1501,6 +1503,11 @@ ldap_parse_rrentry(isc_mem_t *mctx, ldap_entry_t *entry, > return ISC_R_SUCCESS; > > cleanup: > + if (entry != NULL) > + dn = entry->dn; > + if (buf != NULL && str_buf(buf) != NULL) > + data = str_buf(buf); > + log_error_r("failed to parse RR entry: dn '%s': data '%s'", dn, data); > return result; > } > > @@ -1539,7 +1546,6 @@ ldapdb_nodelist_get(isc_mem_t *mctx, ldap_instance_t *ldap_inst, dns_name_t *nam > dns_name_init(&node_name, NULL); > if (dn_to_dnsname(mctx, entry->dn, &node_name, NULL) > != ISC_R_SUCCESS) { > - log_error("Failed to parse dn %s", entry->dn); I prefer to keep this error in place. Otherwise "bad" DNs will be silently skipped. > continue; > } > > @@ -1551,7 +1557,6 @@ ldapdb_nodelist_get(isc_mem_t *mctx, ldap_instance_t *ldap_inst, dns_name_t *nam > string, &node->rdatalist); > } > if (result != ISC_R_SUCCESS) { > - log_error("Failed to parse RR entry (%s)", str_buf(string)); > /* node cleaning */ > dns_name_reset(&node->owner); > ldapdb_rdatalist_destroy(mctx, &node->rdatalist); > @@ -1609,11 +1614,9 @@ ldapdb_rdatalist_get(isc_mem_t *mctx, ldap_instance_t *ldap_inst, dns_name_t *na > for (entry = HEAD(ldap_qresult->ldap_entries); > entry != NULL; > entry = NEXT(entry, link)) { > - if (ldap_parse_rrentry(mctx, entry, ldap_qresult, > - origin, ldap_inst->fake_mname, > - string, rdatalist) != ISC_R_SUCCESS ) { > - log_error("Failed to parse RR entry (%s)", str_buf(string)); > - } > + CHECK(ldap_parse_rrentry(mctx, entry, ldap_qresult, > + origin, ldap_inst->fake_mname, > + string, rdatalist)); > } > > if (!EMPTY(*rdatalist)) { > -- > 1.7.11.4 > -- Adam Tkac, Red Hat, Inc. From atkac at redhat.com Mon Sep 24 13:16:38 2012 From: atkac at redhat.com (Adam Tkac) Date: Mon, 24 Sep 2012 15:16:38 +0200 Subject: [Freeipa-devel] [PATCH 0067] Fix error handling in ldap_get_zone_serial() In-Reply-To: <505C6B03.7020809@redhat.com> References: <505C6B03.7020809@redhat.com> Message-ID: <20120924131637.GC15761@redhat.com> On Fri, Sep 21, 2012 at 03:26:27PM +0200, Petr Spacek wrote: > Hello, > > Fix error handling in ldap_get_zone_serial(). > > Successful call of zr_get_zone_ptr() attaches zone. Zone has to be > detached regardless of dns_zone_getserial2() return code. Ack. > From 5c88eed63ec896bfb4fddb3677dc63b0855c0f6a Mon Sep 17 00:00:00 2001 > From: Petr Spacek > Date: Fri, 21 Sep 2012 15:16:45 +0200 > Subject: [PATCH] Fix error handling in ldap_get_zone_serial(). > > Successful call of zr_get_zone_ptr() attaches zone. Zone has to be > detached regardless of dns_zone_getserial2() return code. > > Signed-off-by: Petr Spacek > --- > src/ldap_helper.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/src/ldap_helper.c b/src/ldap_helper.c > index 2245cb982f26eab165a327b4ad72046f9eb4024e..ec20f9bc36953c2f73ef9ff3d5973fe866cb9b62 100644 > --- a/src/ldap_helper.c > +++ b/src/ldap_helper.c > @@ -2964,9 +2964,10 @@ ldap_get_zone_serial(ldap_instance_t *inst, dns_name_t *zone_name, > > CHECK(zr_get_zone_ptr(inst->zone_register, zone_name, &zone)); > CHECK(dns_zone_getserial2(zone, serial)); > - dns_zone_detach(&zone); > > cleanup: > + if (zone != NULL) > + dns_zone_detach(&zone); > return result; > } > > -- > 1.7.11.4 > -- Adam Tkac, Red Hat, Inc. From pspacek at redhat.com Mon Sep 24 13:21:23 2012 From: pspacek at redhat.com (Petr Spacek) Date: Mon, 24 Sep 2012 15:21:23 +0200 Subject: [Freeipa-devel] [PATCH 0063] Notify DNS slaves if zone serial number modification was detected. In-Reply-To: <20120924130939.GA15761@redhat.com> References: <50571DAA.5030506@redhat.com> <20120924130939.GA15761@redhat.com> Message-ID: <50605E53.2070604@redhat.com> On 09/24/2012 03:09 PM, Adam Tkac wrote: > On Mon, Sep 17, 2012 at 02:55:06PM +0200, Petr Spacek wrote: >> Hello, >> >> this patch adds missing notification to DNS slaves if zone serial >> number modification was detected. > > Hi, > > please check my comment below. > >> From eb8d7fc0c02e03b9c7c90e497225536c449fab1c Mon Sep 17 00:00:00 2001 >> From: Petr Spacek >> Date: Mon, 17 Sep 2012 14:29:45 +0200 >> Subject: [PATCH] Notify DNS slaves if zone serial number modification was >> detected. >> >> Signed-off-by: Petr Spacek >> --- >> src/ldap_helper.c | 22 ++++++++++++++-------- >> 1 file changed, 14 insertions(+), 8 deletions(-) >> >> diff --git a/src/ldap_helper.c b/src/ldap_helper.c >> index 658b960f50b461fa602edf51e955f3bdd4769e1d..d22e8714df57edaad4cf45e6cc26ec0dbbd59108 100644 >> --- a/src/ldap_helper.c >> +++ b/src/ldap_helper.c >> @@ -1035,9 +1035,10 @@ ldap_parse_zoneentry(ldap_entry_t *entry, ldap_instance_t *inst) >> isc_task_t *task = inst->task; >> isc_uint32_t ldap_serial; >> isc_uint32_t zr_serial; /* SOA serial value from in-memory zone register */ >> - unsigned char ldap_digest[RDLIST_DIGESTLENGTH]; >> + unsigned char ldap_digest[RDLIST_DIGESTLENGTH] = {0}; >> unsigned char *zr_digest = NULL; >> ldapdb_rdatalist_t rdatalist; >> + isc_boolean_t zone_dynamic = ISC_FALSE; >> >> REQUIRE(entry != NULL); >> REQUIRE(inst != NULL); >> @@ -1131,13 +1132,13 @@ ldap_parse_zoneentry(ldap_entry_t *entry, ldap_instance_t *inst) >> && result != DNS_R_DYNAMIC && result != DNS_R_CONTINUE) >> goto cleanup; >> >> + zone_dynamic = (result == DNS_R_DYNAMIC); >> result = ISC_R_SUCCESS; >> >> /* initialize serial in zone register and always increment serial >> * for a new zone (typically after BIND start) >> * - the zone was possibly changed in meanwhile */ >> if (publish) { >> - memset(ldap_digest, 0, RDLIST_DIGESTLENGTH); >> CHECK(ldap_get_zone_serial(inst, &name, &ldap_serial)); >> CHECK(zr_set_zone_serial_digest(inst->zone_register, &name, ldap_serial, >> ldap_digest)); >> @@ -1154,23 +1155,28 @@ ldap_parse_zoneentry(ldap_entry_t *entry, ldap_instance_t *inst) >> * 3c) The old and new serials are same: autoincrement only if something >> * else was changed. >> */ >> + CHECK(ldap_get_zone_serial(inst, &name, &ldap_serial)); >> + CHECK(zr_get_zone_serial_digest(inst->zone_register, &name, &zr_serial, >> + &zr_digest)); >> if (inst->serial_autoincrement) { >> - CHECK(ldap_get_zone_serial(inst, &name, &ldap_serial)); >> CHECK(ldapdb_rdatalist_get(inst->mctx, inst, &name, >> &name, &rdatalist)); >> CHECK(rdatalist_digest(inst->mctx, &rdatalist, ldap_digest)); >> >> - CHECK(zr_get_zone_serial_digest(inst->zone_register, &name, &zr_serial, >> - &zr_digest)); >> if (ldap_serial == zr_serial) { >> /* serials are same - increment only if something was changed */ >> if (memcmp(zr_digest, ldap_digest, RDLIST_DIGESTLENGTH) != 0) >> CHECK(soa_serial_increment(inst->mctx, inst, &name)); >> - } else { /* serial in LDAP was changed - update zone register */ >> - CHECK(zr_set_zone_serial_digest(inst->zone_register, &name, >> - ldap_serial, ldap_digest)); >> } >> } >> + if (ldap_serial != zr_serial) { >> + /* serial in LDAP was changed - update zone register */ >> + CHECK(zr_set_zone_serial_digest(inst->zone_register, &name, >> + ldap_serial, ldap_digest)); >> + >> + if (zone_dynamic) >> + dns_zone_notify(zone); > > This if() doesn't seem fully correct to me. Although in most FreeIPA scenarios > it will work, consider situation when someone disables DDNS updates for zone and > modifies records in LDAP. In this case zone_dynamic is FALSE but SOA serial > changes. > > I would recommend to simpy send notify every time when serial changes. Two lines above start of the patch is following code: result = dns_zone_load(zone); if (result != ISC_R_SUCCESS && result != DNS_R_UPTODATE && result != DNS_R_DYNAMIC && result != DNS_R_CONTINUE) goto cleanup; zone_dynamic = (result == DNS_R_DYNAMIC); For non-dynamic zones "dns_zone_load()" sends notifies. Zone_dynamic condition only prevents doubled dns_zone_notify() calls for non-dynamic zones. I can remove this condition if doubled calls are not problem. Petr^2 Spacek > > Regards, Adam > >> + } >> >> cleanup: >> if (unlock) >> -- >> 1.7.11.4 >> > > From pspacek at redhat.com Mon Sep 24 13:27:13 2012 From: pspacek at redhat.com (Petr Spacek) Date: Mon, 24 Sep 2012 15:27:13 +0200 Subject: [Freeipa-devel] [PATCH 0064] Improve log message about improperly formated Resource Records In-Reply-To: <20120924131517.GB15761@redhat.com> References: <50573CC0.8050308@redhat.com> <20120924131517.GB15761@redhat.com> Message-ID: <50605FB1.7030908@redhat.com> On 09/24/2012 03:15 PM, Adam Tkac wrote: > On Mon, Sep 17, 2012 at 05:07:44PM +0200, Petr Spacek wrote: >> Hello, >> >> this patch adds DN to log message about improperly formated Resource Records. > > Hi, > > please check my comment below, otherwise ack. > > Regards, Adam > >> From d36ae54c593c33a45ef9936720357ff7de30c8b5 Mon Sep 17 00:00:00 2001 >> From: Petr Spacek >> Date: Mon, 17 Sep 2012 17:01:41 +0200 >> Subject: [PATCH] Improve log message about improperly formated Resource >> Records. >> >> Signed-off-by: Petr Spacek >> --- >> src/ldap_helper.c | 17 ++++++++++------- >> 1 file changed, 10 insertions(+), 7 deletions(-) >> >> diff --git a/src/ldap_helper.c b/src/ldap_helper.c >> index d22e8714df57edaad4cf45e6cc26ec0dbbd59108..2245cb982f26eab165a327b4ad72046f9eb4024e 100644 >> --- a/src/ldap_helper.c >> +++ b/src/ldap_helper.c >> @@ -1473,6 +1473,8 @@ ldap_parse_rrentry(isc_mem_t *mctx, ldap_entry_t *entry, >> dns_rdata_t *rdata = NULL; >> dns_rdatalist_t *rdlist = NULL; >> ldap_attribute_t *attr; >> + const char *dn = ""; >> + const char *data = ""; >> >> result = add_soa_record(mctx, qresult, origin, entry, >> rdatalist, fake_mname); >> @@ -1501,6 +1503,11 @@ ldap_parse_rrentry(isc_mem_t *mctx, ldap_entry_t *entry, >> return ISC_R_SUCCESS; >> >> cleanup: >> + if (entry != NULL) >> + dn = entry->dn; >> + if (buf != NULL && str_buf(buf) != NULL) >> + data = str_buf(buf); >> + log_error_r("failed to parse RR entry: dn '%s': data '%s'", dn, data); >> return result; >> } >> >> @@ -1539,7 +1546,6 @@ ldapdb_nodelist_get(isc_mem_t *mctx, ldap_instance_t *ldap_inst, dns_name_t *nam >> dns_name_init(&node_name, NULL); >> if (dn_to_dnsname(mctx, entry->dn, &node_name, NULL) >> != ISC_R_SUCCESS) { >> - log_error("Failed to parse dn %s", entry->dn); > > I prefer to keep this error in place. Otherwise "bad" DNs will be silently > skipped. There is another log_error_r() call in dn_to_dnsname(). Actually old code logs error two times and the message from dn_to_dnsname() is more verbose. Petr^2 Spacek > >> continue; >> } >> >> @@ -1551,7 +1557,6 @@ ldapdb_nodelist_get(isc_mem_t *mctx, ldap_instance_t *ldap_inst, dns_name_t *nam >> string, &node->rdatalist); >> } >> if (result != ISC_R_SUCCESS) { >> - log_error("Failed to parse RR entry (%s)", str_buf(string)); >> /* node cleaning */ >> dns_name_reset(&node->owner); >> ldapdb_rdatalist_destroy(mctx, &node->rdatalist); >> @@ -1609,11 +1614,9 @@ ldapdb_rdatalist_get(isc_mem_t *mctx, ldap_instance_t *ldap_inst, dns_name_t *na >> for (entry = HEAD(ldap_qresult->ldap_entries); >> entry != NULL; >> entry = NEXT(entry, link)) { >> - if (ldap_parse_rrentry(mctx, entry, ldap_qresult, >> - origin, ldap_inst->fake_mname, >> - string, rdatalist) != ISC_R_SUCCESS ) { >> - log_error("Failed to parse RR entry (%s)", str_buf(string)); >> - } >> + CHECK(ldap_parse_rrentry(mctx, entry, ldap_qresult, >> + origin, ldap_inst->fake_mname, >> + string, rdatalist)); >> } >> >> if (!EMPTY(*rdatalist)) { >> -- >> 1.7.11.4 >> > > From atkac at redhat.com Mon Sep 24 13:41:55 2012 From: atkac at redhat.com (Adam Tkac) Date: Mon, 24 Sep 2012 15:41:55 +0200 Subject: [Freeipa-devel] [PATCH 0063] Notify DNS slaves if zone serial number modification was detected. In-Reply-To: <50605E53.2070604@redhat.com> References: <50571DAA.5030506@redhat.com> <20120924130939.GA15761@redhat.com> <50605E53.2070604@redhat.com> Message-ID: <20120924134153.GA16545@redhat.com> On Mon, Sep 24, 2012 at 03:21:23PM +0200, Petr Spacek wrote: > On 09/24/2012 03:09 PM, Adam Tkac wrote: > >On Mon, Sep 17, 2012 at 02:55:06PM +0200, Petr Spacek wrote: > >>Hello, > >> > >>this patch adds missing notification to DNS slaves if zone serial > >>number modification was detected. > > > >Hi, > > > >please check my comment below. > > > >> From eb8d7fc0c02e03b9c7c90e497225536c449fab1c Mon Sep 17 00:00:00 2001 > >>From: Petr Spacek > >>Date: Mon, 17 Sep 2012 14:29:45 +0200 > >>Subject: [PATCH] Notify DNS slaves if zone serial number modification was > >> detected. > >> > >>Signed-off-by: Petr Spacek > >>--- > >> src/ldap_helper.c | 22 ++++++++++++++-------- > >> 1 file changed, 14 insertions(+), 8 deletions(-) > >> > >>diff --git a/src/ldap_helper.c b/src/ldap_helper.c > >>index 658b960f50b461fa602edf51e955f3bdd4769e1d..d22e8714df57edaad4cf45e6cc26ec0dbbd59108 100644 > >>--- a/src/ldap_helper.c > >>+++ b/src/ldap_helper.c > >>@@ -1035,9 +1035,10 @@ ldap_parse_zoneentry(ldap_entry_t *entry, ldap_instance_t *inst) > >> isc_task_t *task = inst->task; > >> isc_uint32_t ldap_serial; > >> isc_uint32_t zr_serial; /* SOA serial value from in-memory zone register */ > >>- unsigned char ldap_digest[RDLIST_DIGESTLENGTH]; > >>+ unsigned char ldap_digest[RDLIST_DIGESTLENGTH] = {0}; > >> unsigned char *zr_digest = NULL; > >> ldapdb_rdatalist_t rdatalist; > >>+ isc_boolean_t zone_dynamic = ISC_FALSE; > >> > >> REQUIRE(entry != NULL); > >> REQUIRE(inst != NULL); > >>@@ -1131,13 +1132,13 @@ ldap_parse_zoneentry(ldap_entry_t *entry, ldap_instance_t *inst) > >> && result != DNS_R_DYNAMIC && result != DNS_R_CONTINUE) > >> goto cleanup; > >> > >>+ zone_dynamic = (result == DNS_R_DYNAMIC); > >> result = ISC_R_SUCCESS; > >> > >> /* initialize serial in zone register and always increment serial > >> * for a new zone (typically after BIND start) > >> * - the zone was possibly changed in meanwhile */ > >> if (publish) { > >>- memset(ldap_digest, 0, RDLIST_DIGESTLENGTH); > >> CHECK(ldap_get_zone_serial(inst, &name, &ldap_serial)); > >> CHECK(zr_set_zone_serial_digest(inst->zone_register, &name, ldap_serial, > >> ldap_digest)); > >>@@ -1154,23 +1155,28 @@ ldap_parse_zoneentry(ldap_entry_t *entry, ldap_instance_t *inst) > >> * 3c) The old and new serials are same: autoincrement only if something > >> * else was changed. > >> */ > >>+ CHECK(ldap_get_zone_serial(inst, &name, &ldap_serial)); > >>+ CHECK(zr_get_zone_serial_digest(inst->zone_register, &name, &zr_serial, > >>+ &zr_digest)); > >> if (inst->serial_autoincrement) { > >>- CHECK(ldap_get_zone_serial(inst, &name, &ldap_serial)); > >> CHECK(ldapdb_rdatalist_get(inst->mctx, inst, &name, > >> &name, &rdatalist)); > >> CHECK(rdatalist_digest(inst->mctx, &rdatalist, ldap_digest)); > >> > >>- CHECK(zr_get_zone_serial_digest(inst->zone_register, &name, &zr_serial, > >>- &zr_digest)); > >> if (ldap_serial == zr_serial) { > >> /* serials are same - increment only if something was changed */ > >> if (memcmp(zr_digest, ldap_digest, RDLIST_DIGESTLENGTH) != 0) > >> CHECK(soa_serial_increment(inst->mctx, inst, &name)); > >>- } else { /* serial in LDAP was changed - update zone register */ > >>- CHECK(zr_set_zone_serial_digest(inst->zone_register, &name, > >>- ldap_serial, ldap_digest)); > >> } > >> } > >>+ if (ldap_serial != zr_serial) { > >>+ /* serial in LDAP was changed - update zone register */ > >>+ CHECK(zr_set_zone_serial_digest(inst->zone_register, &name, > >>+ ldap_serial, ldap_digest)); > >>+ > >>+ if (zone_dynamic) > >>+ dns_zone_notify(zone); > > > >This if() doesn't seem fully correct to me. Although in most FreeIPA scenarios > >it will work, consider situation when someone disables DDNS updates for zone and > >modifies records in LDAP. In this case zone_dynamic is FALSE but SOA serial > >changes. > > > >I would recommend to simpy send notify every time when serial changes. > > Two lines above start of the patch is following code: > > result = dns_zone_load(zone); > if (result != ISC_R_SUCCESS && result != DNS_R_UPTODATE > && result != DNS_R_DYNAMIC && result != DNS_R_CONTINUE) > goto cleanup; > > zone_dynamic = (result == DNS_R_DYNAMIC); > > For non-dynamic zones "dns_zone_load()" sends notifies. Zone_dynamic > condition only prevents doubled dns_zone_notify() calls for > non-dynamic zones. I can remove this condition if doubled calls are > not problem. Ah, good point. Then ack for the original version. > Petr^2 Spacek > > > > >Regards, Adam > > > >>+ } > >> > >> cleanup: > >> if (unlock) > >>-- > >>1.7.11.4 > >> > > > > > -- Adam Tkac, Red Hat, Inc. From atkac at redhat.com Mon Sep 24 13:42:31 2012 From: atkac at redhat.com (Adam Tkac) Date: Mon, 24 Sep 2012 15:42:31 +0200 Subject: [Freeipa-devel] [PATCH 0064] Improve log message about improperly formated Resource Records In-Reply-To: <50605FB1.7030908@redhat.com> References: <50573CC0.8050308@redhat.com> <20120924131517.GB15761@redhat.com> <50605FB1.7030908@redhat.com> Message-ID: <20120924134230.GB16545@redhat.com> On Mon, Sep 24, 2012 at 03:27:13PM +0200, Petr Spacek wrote: > On 09/24/2012 03:15 PM, Adam Tkac wrote: > >On Mon, Sep 17, 2012 at 05:07:44PM +0200, Petr Spacek wrote: > >>Hello, > >> > >>this patch adds DN to log message about improperly formated Resource Records. > > > >Hi, > > > >please check my comment below, otherwise ack. > > > >Regards, Adam > > > >> From d36ae54c593c33a45ef9936720357ff7de30c8b5 Mon Sep 17 00:00:00 2001 > >>From: Petr Spacek > >>Date: Mon, 17 Sep 2012 17:01:41 +0200 > >>Subject: [PATCH] Improve log message about improperly formated Resource > >> Records. > >> > >>Signed-off-by: Petr Spacek > >>--- > >> src/ldap_helper.c | 17 ++++++++++------- > >> 1 file changed, 10 insertions(+), 7 deletions(-) > >> > >>diff --git a/src/ldap_helper.c b/src/ldap_helper.c > >>index d22e8714df57edaad4cf45e6cc26ec0dbbd59108..2245cb982f26eab165a327b4ad72046f9eb4024e 100644 > >>--- a/src/ldap_helper.c > >>+++ b/src/ldap_helper.c > >>@@ -1473,6 +1473,8 @@ ldap_parse_rrentry(isc_mem_t *mctx, ldap_entry_t *entry, > >> dns_rdata_t *rdata = NULL; > >> dns_rdatalist_t *rdlist = NULL; > >> ldap_attribute_t *attr; > >>+ const char *dn = ""; > >>+ const char *data = ""; > >> > >> result = add_soa_record(mctx, qresult, origin, entry, > >> rdatalist, fake_mname); > >>@@ -1501,6 +1503,11 @@ ldap_parse_rrentry(isc_mem_t *mctx, ldap_entry_t *entry, > >> return ISC_R_SUCCESS; > >> > >> cleanup: > >>+ if (entry != NULL) > >>+ dn = entry->dn; > >>+ if (buf != NULL && str_buf(buf) != NULL) > >>+ data = str_buf(buf); > >>+ log_error_r("failed to parse RR entry: dn '%s': data '%s'", dn, data); > >> return result; > >> } > >> > >>@@ -1539,7 +1546,6 @@ ldapdb_nodelist_get(isc_mem_t *mctx, ldap_instance_t *ldap_inst, dns_name_t *nam > >> dns_name_init(&node_name, NULL); > >> if (dn_to_dnsname(mctx, entry->dn, &node_name, NULL) > >> != ISC_R_SUCCESS) { > >>- log_error("Failed to parse dn %s", entry->dn); > > > >I prefer to keep this error in place. Otherwise "bad" DNs will be silently > >skipped. > > There is another log_error_r() call in dn_to_dnsname(). Actually old > code logs error two times and the message from dn_to_dnsname() is > more verbose. Thanks for clarification, ack for the original version. > > > > >> continue; > >> } > >> > >>@@ -1551,7 +1557,6 @@ ldapdb_nodelist_get(isc_mem_t *mctx, ldap_instance_t *ldap_inst, dns_name_t *nam > >> string, &node->rdatalist); > >> } > >> if (result != ISC_R_SUCCESS) { > >>- log_error("Failed to parse RR entry (%s)", str_buf(string)); > >> /* node cleaning */ > >> dns_name_reset(&node->owner); > >> ldapdb_rdatalist_destroy(mctx, &node->rdatalist); > >>@@ -1609,11 +1614,9 @@ ldapdb_rdatalist_get(isc_mem_t *mctx, ldap_instance_t *ldap_inst, dns_name_t *na > >> for (entry = HEAD(ldap_qresult->ldap_entries); > >> entry != NULL; > >> entry = NEXT(entry, link)) { > >>- if (ldap_parse_rrentry(mctx, entry, ldap_qresult, > >>- origin, ldap_inst->fake_mname, > >>- string, rdatalist) != ISC_R_SUCCESS ) { > >>- log_error("Failed to parse RR entry (%s)", str_buf(string)); > >>- } > >>+ CHECK(ldap_parse_rrentry(mctx, entry, ldap_qresult, > >>+ origin, ldap_inst->fake_mname, > >>+ string, rdatalist)); > >> } > >> > >> if (!EMPTY(*rdatalist)) { > >>-- > >>1.7.11.4 > >> > > > > > -- Adam Tkac, Red Hat, Inc. From pspacek at redhat.com Mon Sep 24 13:45:25 2012 From: pspacek at redhat.com (Petr Spacek) Date: Mon, 24 Sep 2012 15:45:25 +0200 Subject: [Freeipa-devel] [PATCH 0069] Fix crash caused by empty zone renaming Message-ID: <506063F5.8090702@redhat.com> Hello, Fix crash caused by empty zone renaming. LDAP query uses LDAP_SCOPE_ONELEVEL scope so original condition in INSIST is incorrect, because zone is not required to have child names (i.e. names other than zone itself). -- Petr^2 Spacek -------------- next part -------------- A non-text attachment was scrubbed... Name: bind-dyndb-ldap-pspacek-0069-Fix-crash-caused-by-empty-zone-renaming.patch Type: text/x-patch Size: 1112 bytes Desc: not available URL: From atkac at redhat.com Mon Sep 24 13:45:08 2012 From: atkac at redhat.com (Adam Tkac) Date: Mon, 24 Sep 2012 15:45:08 +0200 Subject: [Freeipa-devel] [PATCH 0068] Fix "unable to concatenate DNS names during zone_refresh" error In-Reply-To: <50604278.7020705@redhat.com> References: <50604278.7020705@redhat.com> Message-ID: <20120924134507.GC16545@redhat.com> On Mon, Sep 24, 2012 at 01:22:32PM +0200, Petr Spacek wrote: > Hello, > > Fix "unable to concatenate DNS names during zone_refresh" error. > > This fixes zone refresh for cases where concatenation of all > zone names is longer than 255 characters. Ack > From e9764d49e8c3ecd6985cd35d7ef1991ee569a98a Mon Sep 17 00:00:00 2001 > From: Petr Spacek > Date: Mon, 24 Sep 2012 10:39:49 +0200 > Subject: [PATCH] Fix "unable to concatenate DNS names during zone_refresh" > error. > > This fixes zone refresh for cases where concatenation of all > zone names is longer than 255 characters. > > Signed-off-by: Petr Spacek > --- > src/ldap_helper.c | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) > > diff --git a/src/ldap_helper.c b/src/ldap_helper.c > index cbb818e7a7510155804a40bc242d7245b1faea90..373d57ac385d4db7394959c23c511e1734503989 100644 > --- a/src/ldap_helper.c > +++ b/src/ldap_helper.c > @@ -1300,10 +1300,10 @@ refresh_zones_from_ldap(ldap_instance_t *ldap_inst, isc_boolean_t delete_only) > result = dns_rbtnodechain_first(&chain, zr_get_rbt(ldap_inst->zone_register), NULL, NULL); > > while (result == DNS_R_NEWORIGIN || result == ISC_R_SUCCESS) { > - > + dns_name_reset(&aname); > delete = ISC_FALSE; > node = NULL; > - > + > result = dns_rbtnodechain_current(&chain, &fname, &forig, &node); > if (result != ISC_R_SUCCESS) { > if (result != ISC_R_NOTFOUND) > @@ -1315,7 +1315,7 @@ refresh_zones_from_ldap(ldap_instance_t *ldap_inst, isc_boolean_t delete_only) > result = dns_name_concatenate(&fname, &forig, &aname, > aname.buffer); > if (result != ISC_R_SUCCESS) { > - log_error_r("unable to concatenate DNS names" > + log_error_r("unable to concatenate DNS names " > "during zone_refresh"); > goto next; > } > -- > 1.7.11.4 > -- Adam Tkac, Red Hat, Inc. From atkac at redhat.com Mon Sep 24 13:46:35 2012 From: atkac at redhat.com (Adam Tkac) Date: Mon, 24 Sep 2012 15:46:35 +0200 Subject: [Freeipa-devel] [PATCH 0069] Fix crash caused by empty zone renaming In-Reply-To: <506063F5.8090702@redhat.com> References: <506063F5.8090702@redhat.com> Message-ID: <20120924134634.GD16545@redhat.com> On Mon, Sep 24, 2012 at 03:45:25PM +0200, Petr Spacek wrote: > Hello, > > Fix crash caused by empty zone renaming. > > LDAP query uses LDAP_SCOPE_ONELEVEL scope so original condition > in INSIST is incorrect, because zone is not required to have child > names (i.e. names other than zone itself). Ack > From 6d59d4beacf5339aec260bac6bd3c69839efb1f7 Mon Sep 17 00:00:00 2001 > From: Petr Spacek > Date: Mon, 24 Sep 2012 15:34:11 +0200 > Subject: [PATCH] Fix crash caused by empty zone renaming. > > LDAP query uses LDAP_SCOPE_ONELEVEL scope so original condition > in INSIST is incorrect, because zone is not required to have child > names (i.e. names other than zone itself). > > Signed-off-by: Petr Spacek > --- > src/ldap_helper.c | 2 -- > 1 file changed, 2 deletions(-) > > diff --git a/src/ldap_helper.c b/src/ldap_helper.c > index 373d57ac385d4db7394959c23c511e1734503989..33eab72f2b37a792a97a6c85f21a3501d399380c 100644 > --- a/src/ldap_helper.c > +++ b/src/ldap_helper.c > @@ -3093,8 +3093,6 @@ update_zone(isc_task_t *task, isc_event_t *event) > LDAP_SCOPE_ONELEVEL, attrs_record, 0, > "(objectClass=idnsRecord)")); > > - /* LDAP schema requires SOA record (at least) */ > - INSIST(HEAD(ldap_qresult_record->ldap_entries) != NULL); > for (entry_record = HEAD(ldap_qresult_record->ldap_entries); > entry_record != NULL; > entry_record = NEXT(entry_record, link)) { > -- > 1.7.11.4 > -- Adam Tkac, Red Hat, Inc. From abokovoy at redhat.com Mon Sep 24 14:01:25 2012 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Mon, 24 Sep 2012 17:01:25 +0300 Subject: [Freeipa-devel] [PATCH] 0078 ipa group-show does not list its members Message-ID: <20120924140125.GB17454@redhat.com> Hi, small patch, to make sure external members are listed when 'ipa group-show' is called. https://fedorahosted.org/freeipa/ticket/2975 -- / Alexander Bokovoy -------------- next part -------------- >From 98a34772855d79cbaf8555809975453fd112b4cf Mon Sep 17 00:00:00 2001 From: Alexander Bokovoy Date: Mon, 24 Sep 2012 16:57:13 +0300 Subject: [PATCH] Make sure external group members are listed for the external group https://fedorahosted.org/freeipa/ticket/2975 --- ipalib/plugins/group.py | 29 ++++++++++------------------- 1 file changed, 10 insertions(+), 19 deletions(-) diff --git a/ipalib/plugins/group.py b/ipalib/plugins/group.py index 3775056a12400ddc236bf5c12ff862731f699431..f1e34bd56fc2427e2e9f60da89cab731021e1db0 100644 --- a/ipalib/plugins/group.py +++ b/ipalib/plugins/group.py @@ -162,6 +162,13 @@ class group(LDAPObject): api.register(group) +ipaexternalmember_param = Str('ipaexternalmember*', + cli_name='external', + label=_('External member'), + doc=_('comma-separated SIDs of members of a trusted domain'), + csv=True, + flags=['no_create', 'no_update', 'no_search'], + ) class group_add(LDAPCreate): __doc__ = _('Create a new group.') @@ -335,22 +342,14 @@ api.register(group_find) class group_show(LDAPRetrieve): __doc__ = _('Display information about a named group.') - + has_output_params = LDAPRetrieve.has_output_params + (ipaexternalmember_param,) api.register(group_show) class group_add_member(LDAPAddMember): __doc__ = _('Add members to a group.') - takes_options = ( - Str('ipaexternalmember*', - cli_name='external', - label=_('External member'), - doc=_('comma-separated SIDs of members of a trusted domain'), - csv=True, - flags=['no_create', 'no_update', 'no_search'], - ), - ) + takes_options = (ipaexternalmember_param,) def post_callback(self, ldap, completed, failed, dn, entry_attrs, *keys, **options): assert isinstance(dn, DN) @@ -390,15 +389,7 @@ api.register(group_add_member) class group_remove_member(LDAPRemoveMember): __doc__ = _('Remove members from a group.') - takes_options = ( - Str('ipaexternalmember*', - cli_name='external', - label=_('External member'), - doc=_('comma-separated SIDs of members of a trusted domain'), - csv=True, - flags=['no_create', 'no_update', 'no_search'], - ), - ) + takes_options = (ipaexternalmember_param,) def pre_callback(self, ldap, dn, found, not_found, *keys, **options): assert isinstance(dn, DN) -- 1.7.12 From atkac at redhat.com Mon Sep 24 14:20:35 2012 From: atkac at redhat.com (Adam Tkac) Date: Mon, 24 Sep 2012 16:20:35 +0200 Subject: [Freeipa-devel] [PATCH 0061] Add missing DNS view attach/detach to LDAP instance management code In-Reply-To: <5051C581.9060209@redhat.com> References: <5051C581.9060209@redhat.com> Message-ID: <20120924142034.GA16999@redhat.com> On Thu, Sep 13, 2012 at 01:37:37PM +0200, Petr Spacek wrote: > Hello, > > Add missing DNS view attach/detach to LDAP instance management code. > This fixes race condition in BIND shutdown after SIGINT: > - failing assert caused by use-after-free in dns_zt_find(): > (((zt) != ((void *)0)) && (((const isc__magic_t *)(zt))->magic > == ((('Z') << 24 | ('T') << 16 | ('b') << 8 | ('l'))))) Ack. > From cc612198a0b7d662557a7c4f71732135e8f43025 Mon Sep 17 00:00:00 2001 > From: Petr Spacek > Date: Thu, 13 Sep 2012 13:08:36 +0200 > Subject: [PATCH] Add missing DNS view attach/detach to LDAP instance > management code. This fixes race condition in BIND shutdown > after SIGINT: - failing assert caused by use-after-free in > dns_zt_find(): (((zt) != ((void *)0)) && (((const > isc__magic_t *)(zt))->magic == ((('Z') << 24 | ('T') << 16 > | ('b') << 8 | ('l'))))) > > Signed-off-by: Petr Spacek > --- > src/ldap_helper.c | 9 ++++----- > 1 file changed, 4 insertions(+), 5 deletions(-) > > diff --git a/src/ldap_helper.c b/src/ldap_helper.c > index 3b49de809738fef18cae10c38fd3d9c33eef5141..658b960f50b461fa602edf51e955f3bdd4769e1d 100644 > --- a/src/ldap_helper.c > +++ b/src/ldap_helper.c > @@ -333,6 +333,7 @@ new_ldap_instance(isc_mem_t *mctx, const char *db_name, > unsigned int i; > isc_result_t result; > ldap_instance_t *ldap_inst; > + dns_view_t *view = NULL; > ld_string_t *auth_method_str = NULL; > setting_t ldap_settings[] = { > { "uri", no_default_string }, > @@ -369,10 +370,9 @@ new_ldap_instance(isc_mem_t *mctx, const char *db_name, > > isc_mem_attach(mctx, &ldap_inst->mctx); > ldap_inst->db_name = db_name; > - ldap_inst->view = dns_dyndb_get_view(dyndb_args); > + view = dns_dyndb_get_view(dyndb_args); > + dns_view_attach(view, &ldap_inst->view); > ldap_inst->zmgr = dns_dyndb_get_zonemgr(dyndb_args); > - /* commented out for now, cause named to hang */ > - //dns_view_attach(view, &ldap_inst->view); > > CHECK(zr_create(mctx, &ldap_inst->zone_register)); > > @@ -616,8 +616,7 @@ destroy_ldap_instance(ldap_instance_t **ldap_instp) > str_destroy(&ldap_inst->fake_mname); > str_destroy(&ldap_inst->ldap_hostname); > > - /* commented out for now, causes named to hang */ > - //dns_view_detach(&ldap_inst->view); > + dns_view_detach(&ldap_inst->view); > > DESTROYLOCK(&ldap_inst->kinit_lock); > > -- > 1.7.11.4 > -- Adam Tkac, Red Hat, Inc. From sbose at redhat.com Mon Sep 24 14:33:16 2012 From: sbose at redhat.com (Sumit Bose) Date: Mon, 24 Sep 2012 16:33:16 +0200 Subject: [Freeipa-devel] [PATCH] 0078 ipa group-show does not list its members In-Reply-To: <20120924140125.GB17454@redhat.com> References: <20120924140125.GB17454@redhat.com> Message-ID: <20120924143315.GH6816@localhost.localdomain> On Mon, Sep 24, 2012 at 05:01:25PM +0300, Alexander Bokovoy wrote: > Hi, > > small patch, to make sure external members are listed when 'ipa > group-show' is called. > > https://fedorahosted.org/freeipa/ticket/2975 ACK bye, Sumit > -- > / Alexander Bokovoy From pviktori at redhat.com Mon Sep 24 14:45:12 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Mon, 24 Sep 2012 16:45:12 +0200 Subject: [Freeipa-devel] [PATCH] 0078 ipa group-show does not list its members In-Reply-To: <20120924140125.GB17454@redhat.com> References: <20120924140125.GB17454@redhat.com> Message-ID: <506071F8.9090103@redhat.com> On 09/24/2012 04:01 PM, Alexander Bokovoy wrote: > Hi, > > small patch, to make sure external members are listed when 'ipa > group-show' is called. > > https://fedorahosted.org/freeipa/ticket/2975 This looks good, but it could use some tests. -- Petr? From abokovoy at redhat.com Mon Sep 24 14:57:00 2012 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Mon, 24 Sep 2012 17:57:00 +0300 Subject: [Freeipa-devel] [PATCH] 0078 ipa group-show does not list its members In-Reply-To: <506071F8.9090103@redhat.com> References: <20120924140125.GB17454@redhat.com> <506071F8.9090103@redhat.com> Message-ID: <20120924145700.GE17454@redhat.com> On Mon, 24 Sep 2012, Petr Viktorin wrote: >On 09/24/2012 04:01 PM, Alexander Bokovoy wrote: >>Hi, >> >>small patch, to make sure external members are listed when 'ipa >>group-show' is called. >> >>https://fedorahosted.org/freeipa/ticket/2975 > >This looks good, but it could use some tests. For external membership being printed? -- / Alexander Bokovoy From pviktori at redhat.com Mon Sep 24 15:08:25 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Mon, 24 Sep 2012 17:08:25 +0200 Subject: [Freeipa-devel] [PATCH] 0078 ipa group-show does not list its members In-Reply-To: <20120924145700.GE17454@redhat.com> References: <20120924140125.GB17454@redhat.com> <506071F8.9090103@redhat.com> <20120924145700.GE17454@redhat.com> Message-ID: <50607769.6070900@redhat.com> On 09/24/2012 04:57 PM, Alexander Bokovoy wrote: > On Mon, 24 Sep 2012, Petr Viktorin wrote: >> On 09/24/2012 04:01 PM, Alexander Bokovoy wrote: >>> Hi, >>> >>> small patch, to make sure external members are listed when 'ipa >>> group-show' is called. >>> >>> https://fedorahosted.org/freeipa/ticket/2975 >> >> This looks good, but it could use some tests. > For external membership being printed? > Ah, I misread what this does. Never mind, sorry for the noise. -- Petr? From abokovoy at redhat.com Mon Sep 24 15:11:27 2012 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Mon, 24 Sep 2012 18:11:27 +0300 Subject: [Freeipa-devel] [PATCH] 0078 ipa group-show does not list its members In-Reply-To: <20120924145700.GE17454@redhat.com> References: <20120924140125.GB17454@redhat.com> <506071F8.9090103@redhat.com> <20120924145700.GE17454@redhat.com> Message-ID: <20120924151127.GF17454@redhat.com> On Mon, 24 Sep 2012, Alexander Bokovoy wrote: >On Mon, 24 Sep 2012, Petr Viktorin wrote: >>On 09/24/2012 04:01 PM, Alexander Bokovoy wrote: >>>Hi, >>> >>>small patch, to make sure external members are listed when 'ipa >>>group-show' is called. >>> >>>https://fedorahosted.org/freeipa/ticket/2975 >> >>This looks good, but it could use some tests. >For external membership being printed? The reason I'm asking is because we have a test for incorrect external member in tests/test_xmlrpc/test_group_plugin.py but we don't have any positive tests for the reason that the code actually verifies SIDs as belonging to configured trusted domains. And without trusted domain you can't pass verification, can't see any external member added. And adding trusted domain is done via complex RPC communication which results in ipasam Samba's passdb module creating specific LDAP entries although we could have faked it with injection of those entries into a database for test purposes, -- / Alexander Bokovoy From rcritten at redhat.com Mon Sep 24 15:20:15 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Mon, 24 Sep 2012 11:20:15 -0400 Subject: [Freeipa-devel] [PATCH] 1057 fix syntax error Message-ID: <50607A2F.1070808@redhat.com> Fix syntax error in exception handling in ipa-client-automount. rob -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-rcrit-1057-automount.patch Type: text/x-diff Size: 1085 bytes Desc: not available URL: From pviktori at redhat.com Mon Sep 24 15:20:58 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Mon, 24 Sep 2012 17:20:58 +0200 Subject: [Freeipa-devel] [PATCH] 0078 ipa group-show does not list its members In-Reply-To: <20120924151127.GF17454@redhat.com> References: <20120924140125.GB17454@redhat.com> <506071F8.9090103@redhat.com> <20120924145700.GE17454@redhat.com> <20120924151127.GF17454@redhat.com> Message-ID: <50607A5A.4040407@redhat.com> On 09/24/2012 05:11 PM, Alexander Bokovoy wrote: > On Mon, 24 Sep 2012, Alexander Bokovoy wrote: >> On Mon, 24 Sep 2012, Petr Viktorin wrote: >>> On 09/24/2012 04:01 PM, Alexander Bokovoy wrote: >>>> Hi, >>>> >>>> small patch, to make sure external members are listed when 'ipa >>>> group-show' is called. >>>> >>>> https://fedorahosted.org/freeipa/ticket/2975 >>> >>> This looks good, but it could use some tests. >> For external membership being printed? > The reason I'm asking is because we have a test for incorrect external > member in tests/test_xmlrpc/test_group_plugin.py but we don't have any > positive tests for the reason that the code actually verifies SIDs as > belonging to configured trusted domains. And without trusted domain you > can't pass verification, can't see any external member added. Yes, I didn't realize external members aren't easy to add from a normal testing environment. > And adding trusted domain is done via complex RPC communication which > results in ipasam Samba's passdb module creating specific LDAP entries > although we could have faked it with injection of those entries into > a database for test purposes, That would be nice to have, though I'm not sure this is the right time to do it. Perhaps file a ticket to remind us later? -- Petr? From abokovoy at redhat.com Mon Sep 24 15:35:50 2012 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Mon, 24 Sep 2012 18:35:50 +0300 Subject: [Freeipa-devel] [PATCH] 0078 ipa group-show does not list its members In-Reply-To: <50607A5A.4040407@redhat.com> References: <20120924140125.GB17454@redhat.com> <506071F8.9090103@redhat.com> <20120924145700.GE17454@redhat.com> <20120924151127.GF17454@redhat.com> <50607A5A.4040407@redhat.com> Message-ID: <20120924153550.GG17454@redhat.com> On Mon, 24 Sep 2012, Petr Viktorin wrote: >On 09/24/2012 05:11 PM, Alexander Bokovoy wrote: >>On Mon, 24 Sep 2012, Alexander Bokovoy wrote: >>>On Mon, 24 Sep 2012, Petr Viktorin wrote: >>>>On 09/24/2012 04:01 PM, Alexander Bokovoy wrote: >>>>>Hi, >>>>> >>>>>small patch, to make sure external members are listed when 'ipa >>>>>group-show' is called. >>>>> >>>>>https://fedorahosted.org/freeipa/ticket/2975 >>>> >>>>This looks good, but it could use some tests. >>>For external membership being printed? >>The reason I'm asking is because we have a test for incorrect external >>member in tests/test_xmlrpc/test_group_plugin.py but we don't have any >>positive tests for the reason that the code actually verifies SIDs as >>belonging to configured trusted domains. And without trusted domain you >>can't pass verification, can't see any external member added. > >Yes, I didn't realize external members aren't easy to add from a >normal testing environment. > >>And adding trusted domain is done via complex RPC communication which >>results in ipasam Samba's passdb module creating specific LDAP entries >>although we could have faked it with injection of those entries into >>a database for test purposes, > >That would be nice to have, though I'm not sure this is the right >time to do it. Perhaps file a ticket to remind us later? Yes, please file something to the backlog. It will help when we'll get to IPA-IPA trusts. -- / Alexander Bokovoy From pviktori at redhat.com Mon Sep 24 15:52:01 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Mon, 24 Sep 2012 17:52:01 +0200 Subject: [Freeipa-devel] [PATCH] 0078 ipa group-show does not list its members In-Reply-To: <20120924153550.GG17454@redhat.com> References: <20120924140125.GB17454@redhat.com> <506071F8.9090103@redhat.com> <20120924145700.GE17454@redhat.com> <20120924151127.GF17454@redhat.com> <50607A5A.4040407@redhat.com> <20120924153550.GG17454@redhat.com> Message-ID: <506081A1.7010301@redhat.com> On 09/24/2012 05:35 PM, Alexander Bokovoy wrote: > On Mon, 24 Sep 2012, Petr Viktorin wrote: >> On 09/24/2012 05:11 PM, Alexander Bokovoy wrote: >>> On Mon, 24 Sep 2012, Alexander Bokovoy wrote: >>>> On Mon, 24 Sep 2012, Petr Viktorin wrote: >>>>> On 09/24/2012 04:01 PM, Alexander Bokovoy wrote: >>>>>> Hi, >>>>>> >>>>>> small patch, to make sure external members are listed when 'ipa >>>>>> group-show' is called. >>>>>> >>>>>> https://fedorahosted.org/freeipa/ticket/2975 >>>>> >>>>> This looks good, but it could use some tests. >>>> For external membership being printed? >>> The reason I'm asking is because we have a test for incorrect external >>> member in tests/test_xmlrpc/test_group_plugin.py but we don't have any >>> positive tests for the reason that the code actually verifies SIDs as >>> belonging to configured trusted domains. And without trusted domain you >>> can't pass verification, can't see any external member added. >> >> Yes, I didn't realize external members aren't easy to add from a >> normal testing environment. >> >>> And adding trusted domain is done via complex RPC communication which >>> results in ipasam Samba's passdb module creating specific LDAP entries >>> although we could have faked it with injection of those entries into >>> a database for test purposes, >> >> That would be nice to have, though I'm not sure this is the right time >> to do it. Perhaps file a ticket to remind us later? > Yes, please file something to the backlog. It will help when we'll get > to IPA-IPA trusts. > https://fedorahosted.org/freeipa/ticket/3115 -- Petr? From pspacek at redhat.com Mon Sep 24 15:59:15 2012 From: pspacek at redhat.com (Petr Spacek) Date: Mon, 24 Sep 2012 17:59:15 +0200 Subject: [Freeipa-devel] [PATCH 0060] Fix zone delete in ldap_zone_delete2() In-Reply-To: <20120924130035.GA15537@redhat.com> References: <5051C540.1030702@redhat.com> <20120924130035.GA15537@redhat.com> Message-ID: <50608353.2030409@redhat.com> On 09/24/2012 03:00 PM, Adam Tkac wrote: > On Thu, Sep 13, 2012 at 01:36:32PM +0200, Petr Spacek wrote: >> Hello, >> >> Fix zone delete in ldap_zone_delete2(). This fixes two race >> conditions during BIND reload: >> >> - failing assert in destroy_ldap_connection() DESTROYLOCK: >> ((pthread_mutex_destroy(&ldap_conn->lock) == 0) ? 0 : 34) == 0 >> >> - use-after-free in call: >> ldap_cache_enabled(cache=0xdededededededede) > > Ack. > > When pushing, please consider if "race-condition" is good description. From my > point of view better is "fix extraction of zone FQDN when destroing LDAP > instance" or something like that. > > Regards, Adam Pushed with amended commit message: e6286d061d12a604fa8c4c3eb282df7ec17b4cbe Petr^2 Spacek From pspacek at redhat.com Mon Sep 24 15:59:32 2012 From: pspacek at redhat.com (Petr Spacek) Date: Mon, 24 Sep 2012 17:59:32 +0200 Subject: [Freeipa-devel] [PATCH 0054] Allow BIND to start if LDAP connection times out In-Reply-To: <20120924121913.GA15023@redhat.com> References: <503CB2F9.7080603@redhat.com> <20120924121913.GA15023@redhat.com> Message-ID: <50608364.2070001@redhat.com> On 09/24/2012 02:19 PM, Adam Tkac wrote: > On Tue, Aug 28, 2012 at 02:00:57PM +0200, Petr Spacek wrote: >> Hello, >> >> this patch allows BIND to start if LDAP connection times out. BIND >> will reconnect in same way as after "connection refused" errors. >> >> The patch closes https://fedorahosted.org/bind-dyndb-ldap/ticket/84 . > > Ack Pushed to master: 6049a88729368082480e3fb76289670912e0217a Petr^2 Spacek From pspacek at redhat.com Mon Sep 24 15:59:46 2012 From: pspacek at redhat.com (Petr Spacek) Date: Mon, 24 Sep 2012 17:59:46 +0200 Subject: [Freeipa-devel] [PATCH 0062] Prevent memory read outside allocated space in str_alloc() In-Reply-To: <20120924120807.GA14825@redhat.com> References: <5052EFF0.1040805@redhat.com> <20120924120807.GA14825@redhat.com> Message-ID: <50608372.9000001@redhat.com> On 09/24/2012 02:08 PM, Adam Tkac wrote: > On Fri, Sep 14, 2012 at 10:50:56AM +0200, Petr Spacek wrote: >> Hello, >> >> Prevent memory read outside allocated space in str_alloc(). >> >> Found by Valgrind during nsupdate stress test. > > Ack Pushed to master: a12a5308d1cccd8bd225036f620f0e049a0fc5f5 Petr^2 Spacek From pspacek at redhat.com Mon Sep 24 16:00:08 2012 From: pspacek at redhat.com (Petr Spacek) Date: Mon, 24 Sep 2012 18:00:08 +0200 Subject: [Freeipa-devel] [PATCH 0063] Notify DNS slaves if zone serial number modification was detected. In-Reply-To: <20120924134153.GA16545@redhat.com> References: <50571DAA.5030506@redhat.com> <20120924130939.GA15761@redhat.com> <50605E53.2070604@redhat.com> <20120924134153.GA16545@redhat.com> Message-ID: <50608388.9080805@redhat.com> On 09/24/2012 03:41 PM, Adam Tkac wrote: > On Mon, Sep 24, 2012 at 03:21:23PM +0200, Petr Spacek wrote: >> On 09/24/2012 03:09 PM, Adam Tkac wrote: >>> On Mon, Sep 17, 2012 at 02:55:06PM +0200, Petr Spacek wrote: >>>> Hello, >>>> >>>> this patch adds missing notification to DNS slaves if zone serial >>>> number modification was detected. >>> >>> Hi, >>> >>> please check my comment below. >>> >>>> From eb8d7fc0c02e03b9c7c90e497225536c449fab1c Mon Sep 17 00:00:00 2001 >>>> From: Petr Spacek >>>> Date: Mon, 17 Sep 2012 14:29:45 +0200 >>>> Subject: [PATCH] Notify DNS slaves if zone serial number modification was >>>> detected. >>>> >>>> Signed-off-by: Petr Spacek >>>> --- >>>> src/ldap_helper.c | 22 ++++++++++++++-------- >>>> 1 file changed, 14 insertions(+), 8 deletions(-) >>>> >>>> diff --git a/src/ldap_helper.c b/src/ldap_helper.c >>>> index 658b960f50b461fa602edf51e955f3bdd4769e1d..d22e8714df57edaad4cf45e6cc26ec0dbbd59108 100644 >>>> --- a/src/ldap_helper.c >>>> +++ b/src/ldap_helper.c >>>> @@ -1035,9 +1035,10 @@ ldap_parse_zoneentry(ldap_entry_t *entry, ldap_instance_t *inst) >>>> isc_task_t *task = inst->task; >>>> isc_uint32_t ldap_serial; >>>> isc_uint32_t zr_serial; /* SOA serial value from in-memory zone register */ >>>> - unsigned char ldap_digest[RDLIST_DIGESTLENGTH]; >>>> + unsigned char ldap_digest[RDLIST_DIGESTLENGTH] = {0}; >>>> unsigned char *zr_digest = NULL; >>>> ldapdb_rdatalist_t rdatalist; >>>> + isc_boolean_t zone_dynamic = ISC_FALSE; >>>> >>>> REQUIRE(entry != NULL); >>>> REQUIRE(inst != NULL); >>>> @@ -1131,13 +1132,13 @@ ldap_parse_zoneentry(ldap_entry_t *entry, ldap_instance_t *inst) >>>> && result != DNS_R_DYNAMIC && result != DNS_R_CONTINUE) >>>> goto cleanup; >>>> >>>> + zone_dynamic = (result == DNS_R_DYNAMIC); >>>> result = ISC_R_SUCCESS; >>>> >>>> /* initialize serial in zone register and always increment serial >>>> * for a new zone (typically after BIND start) >>>> * - the zone was possibly changed in meanwhile */ >>>> if (publish) { >>>> - memset(ldap_digest, 0, RDLIST_DIGESTLENGTH); >>>> CHECK(ldap_get_zone_serial(inst, &name, &ldap_serial)); >>>> CHECK(zr_set_zone_serial_digest(inst->zone_register, &name, ldap_serial, >>>> ldap_digest)); >>>> @@ -1154,23 +1155,28 @@ ldap_parse_zoneentry(ldap_entry_t *entry, ldap_instance_t *inst) >>>> * 3c) The old and new serials are same: autoincrement only if something >>>> * else was changed. >>>> */ >>>> + CHECK(ldap_get_zone_serial(inst, &name, &ldap_serial)); >>>> + CHECK(zr_get_zone_serial_digest(inst->zone_register, &name, &zr_serial, >>>> + &zr_digest)); >>>> if (inst->serial_autoincrement) { >>>> - CHECK(ldap_get_zone_serial(inst, &name, &ldap_serial)); >>>> CHECK(ldapdb_rdatalist_get(inst->mctx, inst, &name, >>>> &name, &rdatalist)); >>>> CHECK(rdatalist_digest(inst->mctx, &rdatalist, ldap_digest)); >>>> >>>> - CHECK(zr_get_zone_serial_digest(inst->zone_register, &name, &zr_serial, >>>> - &zr_digest)); >>>> if (ldap_serial == zr_serial) { >>>> /* serials are same - increment only if something was changed */ >>>> if (memcmp(zr_digest, ldap_digest, RDLIST_DIGESTLENGTH) != 0) >>>> CHECK(soa_serial_increment(inst->mctx, inst, &name)); >>>> - } else { /* serial in LDAP was changed - update zone register */ >>>> - CHECK(zr_set_zone_serial_digest(inst->zone_register, &name, >>>> - ldap_serial, ldap_digest)); >>>> } >>>> } >>>> + if (ldap_serial != zr_serial) { >>>> + /* serial in LDAP was changed - update zone register */ >>>> + CHECK(zr_set_zone_serial_digest(inst->zone_register, &name, >>>> + ldap_serial, ldap_digest)); >>>> + >>>> + if (zone_dynamic) >>>> + dns_zone_notify(zone); >>> >>> This if() doesn't seem fully correct to me. Although in most FreeIPA scenarios >>> it will work, consider situation when someone disables DDNS updates for zone and >>> modifies records in LDAP. In this case zone_dynamic is FALSE but SOA serial >>> changes. >>> >>> I would recommend to simpy send notify every time when serial changes. >> >> Two lines above start of the patch is following code: >> >> result = dns_zone_load(zone); >> if (result != ISC_R_SUCCESS && result != DNS_R_UPTODATE >> && result != DNS_R_DYNAMIC && result != DNS_R_CONTINUE) >> goto cleanup; >> >> zone_dynamic = (result == DNS_R_DYNAMIC); >> >> For non-dynamic zones "dns_zone_load()" sends notifies. Zone_dynamic >> condition only prevents doubled dns_zone_notify() calls for >> non-dynamic zones. I can remove this condition if doubled calls are >> not problem. > > Ah, good point. Then ack for the original version. Pushed to master: c7c18b2e565ddf45d3f20cbab9e4cc2828ec9d5f Petr^2 Spacek From pspacek at redhat.com Mon Sep 24 16:00:17 2012 From: pspacek at redhat.com (Petr Spacek) Date: Mon, 24 Sep 2012 18:00:17 +0200 Subject: [Freeipa-devel] [PATCH 0064] Improve log message about improperly formated Resource Records In-Reply-To: <20120924134230.GB16545@redhat.com> References: <50573CC0.8050308@redhat.com> <20120924131517.GB15761@redhat.com> <50605FB1.7030908@redhat.com> <20120924134230.GB16545@redhat.com> Message-ID: <50608391.7080409@redhat.com> On 09/24/2012 03:42 PM, Adam Tkac wrote: > On Mon, Sep 24, 2012 at 03:27:13PM +0200, Petr Spacek wrote: >> On 09/24/2012 03:15 PM, Adam Tkac wrote: >>> On Mon, Sep 17, 2012 at 05:07:44PM +0200, Petr Spacek wrote: >>>> Hello, >>>> >>>> this patch adds DN to log message about improperly formated Resource Records. >>> >>> Hi, >>> >>> please check my comment below, otherwise ack. >>> >>> Regards, Adam Pushed to master: 11f5ee88185681c39e2bee01adff7f33670063b7 Petr^2 Spacek From pspacek at redhat.com Mon Sep 24 16:00:24 2012 From: pspacek at redhat.com (Petr Spacek) Date: Mon, 24 Sep 2012 18:00:24 +0200 Subject: [Freeipa-devel] [PATCH 0067] Fix error handling in ldap_get_zone_serial() In-Reply-To: <20120924131637.GC15761@redhat.com> References: <505C6B03.7020809@redhat.com> <20120924131637.GC15761@redhat.com> Message-ID: <50608398.70002@redhat.com> On 09/24/2012 03:16 PM, Adam Tkac wrote: > On Fri, Sep 21, 2012 at 03:26:27PM +0200, Petr Spacek wrote: >> Hello, >> >> Fix error handling in ldap_get_zone_serial(). >> >> Successful call of zr_get_zone_ptr() attaches zone. Zone has to be >> detached regardless of dns_zone_getserial2() return code. > > Ack. Pushed to master: ba97cf50727ee6435771be204fb32ad66f53f918 Petr^2 Spacek From pspacek at redhat.com Mon Sep 24 16:00:34 2012 From: pspacek at redhat.com (Petr Spacek) Date: Mon, 24 Sep 2012 18:00:34 +0200 Subject: [Freeipa-devel] [PATCH 0061] Add missing DNS view attach/detach to LDAP instance management code In-Reply-To: <20120924142034.GA16999@redhat.com> References: <5051C581.9060209@redhat.com> <20120924142034.GA16999@redhat.com> Message-ID: <506083A2.8010105@redhat.com> On 09/24/2012 04:20 PM, Adam Tkac wrote: > On Thu, Sep 13, 2012 at 01:37:37PM +0200, Petr Spacek wrote: >> Hello, >> >> Add missing DNS view attach/detach to LDAP instance management code. >> This fixes race condition in BIND shutdown after SIGINT: >> - failing assert caused by use-after-free in dns_zt_find(): >> (((zt) != ((void *)0)) && (((const isc__magic_t *)(zt))->magic >> == ((('Z') << 24 | ('T') << 16 | ('b') << 8 | ('l'))))) > > Ack. Pushed to master: d26d61c4add5593ffca4380029748c2d359140c5 Petr^2 Spacek From pspacek at redhat.com Mon Sep 24 16:00:51 2012 From: pspacek at redhat.com (Petr Spacek) Date: Mon, 24 Sep 2012 18:00:51 +0200 Subject: [Freeipa-devel] [PATCH 0069] Fix crash caused by empty zone renaming In-Reply-To: <20120924134634.GD16545@redhat.com> References: <506063F5.8090702@redhat.com> <20120924134634.GD16545@redhat.com> Message-ID: <506083B3.4030802@redhat.com> On 09/24/2012 03:46 PM, Adam Tkac wrote: > On Mon, Sep 24, 2012 at 03:45:25PM +0200, Petr Spacek wrote: >> Hello, >> >> Fix crash caused by empty zone renaming. >> >> LDAP query uses LDAP_SCOPE_ONELEVEL scope so original condition >> in INSIST is incorrect, because zone is not required to have child >> names (i.e. names other than zone itself). > > Ack Pushed to master: 0722253e719e94ba6917756d8ba1a103cd2bb048 Petr^2 Spacek From pspacek at redhat.com Mon Sep 24 16:00:42 2012 From: pspacek at redhat.com (Petr Spacek) Date: Mon, 24 Sep 2012 18:00:42 +0200 Subject: [Freeipa-devel] [PATCH 0068] Fix "unable to concatenate DNS names during zone_refresh" error In-Reply-To: <20120924134507.GC16545@redhat.com> References: <50604278.7020705@redhat.com> <20120924134507.GC16545@redhat.com> Message-ID: <506083AA.4020407@redhat.com> On 09/24/2012 03:45 PM, Adam Tkac wrote: > On Mon, Sep 24, 2012 at 01:22:32PM +0200, Petr Spacek wrote: >> Hello, >> >> Fix "unable to concatenate DNS names during zone_refresh" error. >> >> This fixes zone refresh for cases where concatenation of all >> zone names is longer than 255 characters. > > Ack Pushed to master: 186f4b4cb8ea9e42421e40ac426d0b79e668860c Petr^2 Spacek From mkosek at redhat.com Mon Sep 24 16:01:37 2012 From: mkosek at redhat.com (Martin Kosek) Date: Mon, 24 Sep 2012 18:01:37 +0200 Subject: [Freeipa-devel] [PATCH] 1057 fix syntax error In-Reply-To: <50607A2F.1070808@redhat.com> References: <50607A2F.1070808@redhat.com> Message-ID: <506083E1.9060204@redhat.com> On 09/24/2012 05:20 PM, Rob Crittenden wrote: > Fix syntax error in exception handling in ipa-client-automount. > > rob > Obvious ACK. Pushed to master, ipa-3-0. Martin From ssorce at redhat.com Mon Sep 24 18:40:45 2012 From: ssorce at redhat.com (Simo Sorce) Date: Mon, 24 Sep 2012 14:40:45 -0400 (EDT) Subject: [Freeipa-devel] [PATCH] Simplify get_group_sids In-Reply-To: <1738904500.827729.1348512041694.JavaMail.root@redhat.com> Message-ID: <1791110990.827734.1348512045590.JavaMail.root@redhat.com> This should also give us a slight performance boost as we do not convert the whole SID to a string many times over. Simo. -- Simo Sorce * Red Hat, Inc. * New York -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Simplify-get_group_sids.patch Type: text/x-patch Size: 3424 bytes Desc: not available URL: From sbose at redhat.com Mon Sep 24 19:52:00 2012 From: sbose at redhat.com (Sumit Bose) Date: Mon, 24 Sep 2012 21:52:00 +0200 Subject: [Freeipa-devel] [PATCH] Simplify get_group_sids In-Reply-To: <1791110990.827734.1348512045590.JavaMail.root@redhat.com> References: <1738904500.827729.1348512041694.JavaMail.root@redhat.com> <1791110990.827734.1348512045590.JavaMail.root@redhat.com> Message-ID: <20120924195200.GM6816@localhost.localdomain> On Mon, Sep 24, 2012 at 02:40:45PM -0400, Simo Sorce wrote: > > This should also give us a slight performance boost as we do > not convert the whole SID to a string many times over. > I was digging up the archive URL of my patch related to this posted to the list on July (https://www.redhat.com/archives/freeipa-devel/2012-July/msg00089.html) when I realized that you already reviewed it and gave some comments (https://www.redhat.com/archives/freeipa-devel/2012-July/msg00132.html). For some reasons this mail never made it into my inbox. Since the old patch does not only covers the SID to string conversion but adds other improvements to memory allocation I would like to ask if you can ACK the old one if I fix the issues you mentioend in your comment and drop yours instead? bye, Sumit > Simo. > > -- > Simo Sorce * Red Hat, Inc. * New York From ssorce at redhat.com Mon Sep 24 19:53:15 2012 From: ssorce at redhat.com (Simo Sorce) Date: Mon, 24 Sep 2012 15:53:15 -0400 (EDT) Subject: [Freeipa-devel] [PATCH] Simplify get_group_sids In-Reply-To: <20120924195200.GM6816@localhost.localdomain> Message-ID: <89176914.866626.1348516395237.JavaMail.root@redhat.com> ----- Original Message ----- > On Mon, Sep 24, 2012 at 02:40:45PM -0400, Simo Sorce wrote: > > > > This should also give us a slight performance boost as we do > > not convert the whole SID to a string many times over. > > > > I was digging up the archive URL of my patch related to this posted > to > the list on July > (https://www.redhat.com/archives/freeipa-devel/2012-July/msg00089.html) > when I realized that you already reviewed it and gave some comments > (https://www.redhat.com/archives/freeipa-devel/2012-July/msg00132.html). > For > some reasons this mail never made it into my inbox. > > Since the old patch does not only covers the SID to string conversion > but adds other improvements to memory allocation I would like to ask > if > you can ACK the old one if I fix the issues you mentioend in your > comment and drop yours instead? Yeah feel free to do so. Simo. -- Simo Sorce * Red Hat, Inc. * New York From mkosek at redhat.com Tue Sep 25 06:22:23 2012 From: mkosek at redhat.com (Martin Kosek) Date: Tue, 25 Sep 2012 08:22:23 +0200 Subject: [Freeipa-devel] [PATCH] 0078 ipa group-show does not list its members In-Reply-To: <20120924143315.GH6816@localhost.localdomain> References: <20120924140125.GB17454@redhat.com> <20120924143315.GH6816@localhost.localdomain> Message-ID: <50614D9F.5000400@redhat.com> On 09/24/2012 04:33 PM, Sumit Bose wrote: > On Mon, Sep 24, 2012 at 05:01:25PM +0300, Alexander Bokovoy wrote: >> Hi, >> >> small patch, to make sure external members are listed when 'ipa >> group-show' is called. >> >> https://fedorahosted.org/freeipa/ticket/2975 > > ACK > > bye, > Sumit Pushed to master, ipa-3-0. Martin From mkosek at redhat.com Tue Sep 25 08:42:59 2012 From: mkosek at redhat.com (Martin Kosek) Date: Tue, 25 Sep 2012 10:42:59 +0200 Subject: [Freeipa-devel] [PATCH] 312 Use custom zonemgr for reverse zones Message-ID: <50616E93.80509@redhat.com> When DNS is being installed during ipa-{server,dns,replica}-install, forward and reverse zone is created. However, reverse zone was always created with default zonemgr even when a custom zonemgr was passed to the installer as this functionality was missing in function creating reverse zone. Consolidate functions creating forward and reverse zones to avoid code duplication and errors like this one. Reverse zones are now created with custom zonemgr (when entered by user). https://fedorahosted.org/freeipa/ticket/2790 -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-312-use-custom-zonemgr-for-reverse-zones.patch Type: text/x-patch Size: 10117 bytes Desc: not available URL: From tbabej at redhat.com Tue Sep 25 10:37:01 2012 From: tbabej at redhat.com (Tomas Babej) Date: Tue, 25 Sep 2012 12:37:01 +0200 Subject: [Freeipa-devel] [PATCH 0014] Improve user addition to default group in host-add Message-ID: <5061894D.9010202@redhat.com> Hi, On adding new user, host-add tries to make it a member of default user group. This, however, can raise AlreadyGroupMember when the user is already member of this group due to automember rule or default group configured. This patch makes sure AlreadyGroupMember exception is caught in such cases. https://fedorahosted.org/freeipa/ticket/3097 Tomas -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-tbabej-0014-Improve-user-addition-to-default-group-in-host-add.patch Type: text/x-patch Size: 1657 bytes Desc: not available URL: From mkosek at redhat.com Tue Sep 25 11:54:06 2012 From: mkosek at redhat.com (Martin Kosek) Date: Tue, 25 Sep 2012 13:54:06 +0200 Subject: [Freeipa-devel] [PATCH] 313 Validate SELinux users in config-mod Message-ID: <50619B5E.2070805@redhat.com> config-mod is capable of changing default SELinux user map order and a default SELinux user. Validate the new config values to prevent bogus default SELinux users to be assigned to IPA users. https://fedorahosted.org/freeipa/ticket/2993 --- Note: I removed the previous "validate" construct: - validate = dict(options) - validate.update(entry_attrs) ... as entry_attrs contains both values set via standard options and *attr. Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-313-validate-selinux-users-in-config-mod.patch Type: text/x-patch Size: 8273 bytes Desc: not available URL: From tbabej at redhat.com Tue Sep 25 12:22:26 2012 From: tbabej at redhat.com (Tomas Babej) Date: Tue, 25 Sep 2012 14:22:26 +0200 Subject: [Freeipa-devel] [PATCH 0015] Restrict admins group modifications Message-ID: <5061A202.2090203@redhat.com> Hi, Group-mod command no longer allows --rename and/or --external changes made to the admins group. In such cases, ProtectedEntryError is being raised. https://fedorahosted.org/freeipa/ticket/3098 Tomas -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-tbabej-0015-Restrict-admins-group-modifications.patch Type: text/x-patch Size: 3491 bytes Desc: not available URL: From mkosek at redhat.com Tue Sep 25 12:31:46 2012 From: mkosek at redhat.com (Martin Kosek) Date: Tue, 25 Sep 2012 14:31:46 +0200 Subject: [Freeipa-devel] [PATCH 0015] Restrict admins group modifications In-Reply-To: <5061A202.2090203@redhat.com> References: <5061A202.2090203@redhat.com> Message-ID: <5061A432.3090709@redhat.com> On 09/25/2012 02:22 PM, Tomas Babej wrote: > Hi, > > Group-mod command no longer allows --rename and/or --external > changes made to the admins group. In such cases, ProtectedEntryError > is being raised. > > https://fedorahosted.org/freeipa/ticket/3098 > > Tomas > Just based on a quick glance, I see few issues: 1) I would like to see unit tests for this scenario 2) Some overlooked debug output: + self.log.info(str(keys)) Martin From tbabej at redhat.com Tue Sep 25 12:59:23 2012 From: tbabej at redhat.com (Tomas Babej) Date: Tue, 25 Sep 2012 14:59:23 +0200 Subject: [Freeipa-devel] [PATCH 0015] Restrict admins group modifications In-Reply-To: <5061A432.3090709@redhat.com> References: <5061A202.2090203@redhat.com> <5061A432.3090709@redhat.com> Message-ID: <5061AAAB.7020801@redhat.com> On 09/25/2012 02:31 PM, Martin Kosek wrote: > On 09/25/2012 02:22 PM, Tomas Babej wrote: >> Hi, >> >> Group-mod command no longer allows --rename and/or --external >> changes made to the admins group. In such cases, ProtectedEntryError >> is being raised. >> >> https://fedorahosted.org/freeipa/ticket/3098 >> >> Tomas >> > Just based on a quick glance, I see few issues: > > 1) I would like to see unit tests for this scenario > > 2) Some overlooked debug output: > > + self.log.info(str(keys)) > > Martin I removed the unfortunate debug output and added two unit tests to test the scenarios. Tomas -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-tbabej-0015-2-Restrict-admins-group-modifications.patch Type: text/x-patch Size: 4666 bytes Desc: not available URL: From abokovoy at redhat.com Tue Sep 25 14:30:49 2012 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Tue, 25 Sep 2012 17:30:49 +0300 Subject: [Freeipa-devel] [PATCH] 0079 support creating LDAP control by python-ldap 2.3 (RHEL) and newer versions (Fedora) Message-ID: <20120925143049.GL17454@redhat.com> Hi, I did have bug filed against python-ldap in January and for some reason my patch to accomodate two ways of making LDAP controls was not included in March 2012 when I presented it as part of trusts, but yesterday we found it is really needed for RHEL6 version of python-ldap. Rather than having separate patch, I'd prefer to have both versions supported upstream. The same issue was with Fedora 16 versus Fedora 17. -- / Alexander Bokovoy -------------- next part -------------- >From 9d6f09aadd2bd47660934d0f606e9a326456d6e4 Mon Sep 17 00:00:00 2001 From: Alexander Bokovoy Date: Tue, 25 Sep 2012 17:23:33 +0300 Subject: [PATCH 1/2] Support python-ldap 2.3 way of making LDAP control This strange patch is to accomodate both python-ldap 2.3 and later versions. There was refactoring in python-ldap support for LDAP controls that split base class into two different, changing properties and method signatures. Luckily, we don't use any values passed to encodeControlValue. --- ipaserver/dcerpc.py | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/ipaserver/dcerpc.py b/ipaserver/dcerpc.py index 86cf01dbac9aca21c35d2db65ef4d4c56e313709..ae0738db0de08a6fbc01808a5bd702418fe17778 100644 --- a/ipaserver/dcerpc.py +++ b/ipaserver/dcerpc.py @@ -40,8 +40,12 @@ from samba.ndr import ndr_pack from samba import net import samba import random -import ldap as _ldap from Crypto.Cipher import ARC4 +try: + from ldap.controls import RequestControl as LDAPControl #pylint: disable=F0401 +except: + from ldap.controls import LDAPControl as LDAPControl #pylint: disable=F0401 +import ldap as _ldap __doc__ = _(""" Classes to manage trust joins using DCE-RPC calls @@ -81,13 +85,17 @@ def assess_dcerpc_exception(num=None,message=None): message "%(message)s" (both may be "None")''') % dict(num=num, message=message) return errors.RemoteRetrieveError(reason=reason) -class ExtendedDNControl(_ldap.controls.RequestControl): +class ExtendedDNControl(LDAPControl): + # This class attempts to implement LDAP control that would work + # with both python-ldap 2.4.x and 2.3.x, thus there is mix of properties + # from both worlds and encodeControlValue has default parameter def __init__(self): + self.controlValue = 1 self.controlType = "1.2.840.113556.1.4.529" self.criticality = False self.integerValue = 1 - def encodeControlValue(self): + def encodeControlValue(self, value=None): return '0\x03\x02\x01\x01' class DomainValidator(object): -- 1.7.12 From pviktori at redhat.com Tue Sep 25 14:38:40 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Tue, 25 Sep 2012 16:38:40 +0200 Subject: [Freeipa-devel] [PATCH] 0084 Wait for secure Dogtag ports when starting the pki services Message-ID: <5061C1F0.1050501@redhat.com> Dogtag opens not only the insecure port (8080 or 9180, for d10 or d9 respectively), but also secure ports (8443 or 9443&9444). Wait for them when starting. Part of the fix for https://fedorahosted.org/freeipa/ticket/3084. I found that if we don't wait for the secure port, /ca/admin/ca/getStatus will give a service unavailable error. I'm working on a patch to utilize the new status information. I'll submit it after the updated Dogtag builds get in. -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0084-Wait-for-secure-Dogtag-ports-when-starting-the-pki-s.patch Type: text/x-patch Size: 1363 bytes Desc: not available URL: From abokovoy at redhat.com Tue Sep 25 14:40:57 2012 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Tue, 25 Sep 2012 17:40:57 +0300 Subject: [Freeipa-devel] [PATCH] 0080 rewrite SID comparison to take into account different SID forms Message-ID: <20120925144057.GM17454@redhat.com> Hi, Domain validator code in ipaserver/dcerpc.py verifies that a SID belongs to one of our trusted domains. This verification was expecting that SID is for some resource within trusted domain and ignored the case when it is the SID of the trusted domain, i.e. when sid has form like S-1-5-21-16904141-148189700-2149043814 rather than S-1-5-21-16904141-148189700-2149043814-512 (Domain Admins). The latter is what idrange-add command uses. So comparing SID with SID was done by stripping last component (RID). In case of idrange-add stripping last RID was making a SID that could never compare to a trusted domain SID. Somehow the code worked for me in Fedora and started failing on RHEL6. -- / Alexander Bokovoy -------------- next part -------------- >From 6efa779009b86bca31e95ef3640b179576d21eff Mon Sep 17 00:00:00 2001 From: Alexander Bokovoy Date: Tue, 25 Sep 2012 17:25:42 +0300 Subject: [PATCH 2/2] Change the way SID comparison is done for belonging to trusted domain Fixes trust use on RHEL 6. --- ipaserver/dcerpc.py | 28 ++++++++++++---------------- 1 file changed, 12 insertions(+), 16 deletions(-) diff --git a/ipaserver/dcerpc.py b/ipaserver/dcerpc.py index ae0738db0de08a6fbc01808a5bd702418fe17778..7108bc8235e00b975ec152d1fbf2519f4679b5c6 100644 --- a/ipaserver/dcerpc.py +++ b/ipaserver/dcerpc.py @@ -132,7 +132,8 @@ class DomainValidator(object): (entries, truncated) = self.ldap.find_entries(filter=filter, base_dn=cn_trust, attrs_list=[self.ATTR_TRUSTED_SID, 'dn']) - return entries + result = map (lambda entry: security.dom_sid(entry[1][self.ATTR_TRUSTED_SID][0]), entries) + return result except errors.NotFound, e: return [] @@ -144,17 +145,7 @@ class DomainValidator(object): # Parse sid string to see if it is really in a SID format try: test_sid = security.dom_sid(sid) - except TypeError: - return False - (dom, sid_rid) = test_sid.split() - sid_dom = str(dom) - # Now we have domain prefix of the sid as sid_dom string and can - # analyze it against known prefixes - if sid_dom.find(security.SID_NT_AUTHORITY) != 0: - # Ignore any potential SIDs that are not S-1-5-* - return False - if sid_dom.find(self.sid) == 0: - # A SID from our own domain cannot be treated as trusted domain's SID + except TypeError, e: return False # At this point we have SID_NT_AUTHORITY family SID and really need to # check it against prefixes of domain SIDs we trust to @@ -166,8 +157,11 @@ class DomainValidator(object): return False # We have non-zero list of trusted domains and have to go through them # one by one and check their sids as prefixes - for (dn, domaininfo) in self._domains: - if sid_dom.find(domaininfo[self.ATTR_TRUSTED_SID][0]) == 0: + test_sid_subauths = test_sid.sub_auths + for domsid in self._domains: + sub_auths = domsid.sub_auths + num_auths = min(test_sid.num_auths, domsid.num_auths) + if test_sid_subauths[:num_auths] == sub_auths[:num_auths]: return True return False @@ -401,8 +395,10 @@ class TrustDomainInstance(object): result = retrieve_netlogon_info_2(self, netlogon.NETLOGON_CONTROL_TC_VERIFY, another_domain.info['dns_domain']) - if (unicode(result.trusted_dc_name)[2:] == another_domain.info['dc'] and - result.tc_connection_status == (0, 'WERR_OK')): + if (result and (result.flags and netlogon.NETLOGON_VERIFY_STATUS_RETURNED)): + # netr_LogonControl2Ex() returns non-None result only if overall call + # result was WERR_OK which means verification was correct. + # We only check that it was indeed status for verification process return True return False -- 1.7.12 From abokovoy at redhat.com Tue Sep 25 14:46:58 2012 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Tue, 25 Sep 2012 17:46:58 +0300 Subject: [Freeipa-devel] [PATCH] 0080 rewrite SID comparison to take into account different SID forms In-Reply-To: <20120925144057.GM17454@redhat.com> References: <20120925144057.GM17454@redhat.com> Message-ID: <20120925144658.GN17454@redhat.com> On Tue, 25 Sep 2012, Alexander Bokovoy wrote: > Hi, > > Domain validator code in ipaserver/dcerpc.py verifies that a SID belongs > to one of our trusted domains. This verification was expecting that SID > is for some resource within trusted domain and ignored the case when it > is the SID of the trusted domain, i.e. when sid has form like > S-1-5-21-16904141-148189700-2149043814 rather than > S-1-5-21-16904141-148189700-2149043814-512 (Domain Admins). > > The latter is what idrange-add command uses. The *former*, sigh. :) -- / Alexander Bokovoy From pviktori at redhat.com Tue Sep 25 15:59:27 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Tue, 25 Sep 2012 17:59:27 +0200 Subject: [Freeipa-devel] [PATCH] 0085 Don't use bare except: clauses in ipa-client-install Message-ID: <5061D4DF.90207@redhat.com> https://fedorahosted.org/freeipa/ticket/2941 Simo, do you remember how you configured resolv.conf and when you pressed Ctrl+C? There's a lot of code called from ipa-replica-install that could cause the bug. Here I went for the most obvious places where we ignore KeyboardInterrupt. I will look for others. I'll also do a more thorough audit of the installer code when I work on https://fedorahosted.org/freeipa/ticket/2652. -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0085-Don-t-use-bare-except-clauses-in-ipa-client-install.patch Type: text/x-patch Size: 4439 bytes Desc: not available URL: From mkosek at redhat.com Tue Sep 25 16:13:39 2012 From: mkosek at redhat.com (Martin Kosek) Date: Tue, 25 Sep 2012 18:13:39 +0200 Subject: [Freeipa-devel] [PATCH] 314-315 Limit unindexed searches Message-ID: <5061D833.10204@redhat.com> These 2 patches significantly limit the number of unindexed LDAP searches we do in IPA. I used our unit test suite as a good source of different LDAP searches run by our command suite. Most of the remaining unindexed searches are produced either by our general term search ("ipa service-find TERM") which hit every object parameter and DNS commands (idnsname is not indexed yet). I am thinking about indexing about idnsName as well... Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-314-add-indexes.patch Type: text/x-patch Size: 2326 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-315-unindexed-searche-on-del.patch Type: text/x-patch Size: 1783 bytes Desc: not available URL: From pviktori at redhat.com Wed Sep 26 07:59:56 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Wed, 26 Sep 2012 09:59:56 +0200 Subject: [Freeipa-devel] [PATCH 0014] Improve user addition to default group in host-add In-Reply-To: <5061894D.9010202@redhat.com> References: <5061894D.9010202@redhat.com> Message-ID: <5062B5FC.6000900@redhat.com> On 09/25/2012 12:37 PM, Tomas Babej wrote: > Hi, > > On adding new user, host-add tries to make it a member of default > user group. This, however, can raise AlreadyGroupMember when the > user is already member of this group due to automember rule or > default group configured. This patch makes sure AlreadyGroupMember > exception is caught in such cases. > > https://fedorahosted.org/freeipa/ticket/3097 > > Tomas > > This looks good, but please also add a test for it. -- Petr? From pviktori at redhat.com Wed Sep 26 10:25:44 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Wed, 26 Sep 2012 12:25:44 +0200 Subject: [Freeipa-devel] [PATCH] 313 Validate SELinux users in config-mod In-Reply-To: <50619B5E.2070805@redhat.com> References: <50619B5E.2070805@redhat.com> Message-ID: <5062D828.5000302@redhat.com> On 09/25/2012 01:54 PM, Martin Kosek wrote: > config-mod is capable of changing default SELinux user map order > and a default SELinux user. Validate the new config values to > prevent bogus default SELinux users to be assigned to IPA users. > > https://fedorahosted.org/freeipa/ticket/2993 > > --- > Note: I removed the previous "validate" construct: > - validate = dict(options) > - validate.update(entry_attrs) > ... as entry_attrs contains both values set via standard options and *attr. > > Martin > The patch looks OK, see a comment below. I found strange behavior in validate_selinuxuser. Perhaps it's material for another ticket. This command passes validation: $ ./ipa config_mod --ipaselinuxusermapdefault=unconfined_u:s0-s0:c0.c1023 --ipaselinuxusermaporder='unconfined_u:s0-s0:c0.c1023$xguest_u:s5-s1:c0-c4.c4,c4:?Why is stuff allowed here?' [...] SELinux user map order: unconfined_u:s0-s0:c0.c1023$xguest_u:s5-s1:c0-c4.c3.c8,c4:?Why is stuff allowed here? Default SELinux user: unconfined_u:s0-s0:c0.c1023 PAC type: MS-PAC Obviously extra info should not be allowed. Is "s5-s1" or "c4.c3" valid? Can the first value be higher than the second? AFAIK (I'm not an expert though), MCS doesn't allow dashes, so "c0-c4" should not be allowed. Chains like "c1.c2.c3" also don't look right. > freeipa-mkosek-313-validate-selinux-users-in-config-mod.patch > > > From 296eedc7cfd258b9e5eaf4f182b1a9625f5bf1a1 Mon Sep 17 00:00:00 2001 > From: Martin Kosek > Date: Tue, 25 Sep 2012 13:46:56 +0200 > Subject: [PATCH] Validate SELinux users in config-mod > > config-mod is capable of changing default SELinux user map order > and a default SELinux user. Validate the new config values to > prevent bogus default SELinux users to be assigned to IPA users. > > https://fedorahosted.org/freeipa/ticket/2993 > --- > ipalib/plugins/config.py | 49 +++++++++++++++++++++------------ > tests/test_xmlrpc/test_config_plugin.py | 44 ++++++++++++++++++++++++----- > 2 files changed, 69 insertions(+), 24 deletions(-) > > diff --git a/ipalib/plugins/config.py b/ipalib/plugins/config.py > index e02519d5759f4e4a6d6a7075fe896f8b2e69b451..1c62e0d942231fac442ee2c1f31431003c08e283 100644 > --- a/ipalib/plugins/config.py > +++ b/ipalib/plugins/config.py [...] > + # validate the new user order first > + for user in userlist: > + if not user: > + raise errors.ValidationError(name='ipaselinuxusermaporder', > + error=_('A list of SELinux users delimited by $ expected')) > + This will only catch empty users (i.e. a "$$", or a "$" at beginning or end), right? A specific message like "empty users not allowed" could be more helpful. -- Petr? From pviktori at redhat.com Wed Sep 26 10:32:29 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Wed, 26 Sep 2012 12:32:29 +0200 Subject: [Freeipa-devel] [PATCH] 313 Validate SELinux users in config-mod In-Reply-To: <5062D828.5000302@redhat.com> References: <50619B5E.2070805@redhat.com> <5062D828.5000302@redhat.com> Message-ID: <5062D9BD.2020407@redhat.com> On 09/26/2012 12:25 PM, Petr Viktorin wrote: > > I found strange behavior in validate_selinuxuser. Perhaps it's material > for another ticket. This command passes validation: > > $ ./ipa config_mod > --ipaselinuxusermapdefault=unconfined_u:s0-s0:c0.c1023 > --ipaselinuxusermaporder='unconfined_u:s0-s0:c0.c1023$xguest_u:s5-s1:c0-c4.c4,c4:?Why > is stuff allowed here?' > [...] > SELinux user map order: > unconfined_u:s0-s0:c0.c1023$xguest_u:s5-s1:c0-c4.c3.c8,c4:?Why is stuff > allowed here? > Default SELinux user: unconfined_u:s0-s0:c0.c1023 > PAC type: MS-PAC > > Obviously extra info should not be allowed. > Is "s5-s1" or "c4.c3" valid? Can the first value be higher than the second? > AFAIK (I'm not an expert though), MCS doesn't allow dashes, so "c0-c4" > should not be allowed. Chains like "c1.c2.c3" also don't look right. ... Also, the MLS/MCS numeric limits are not enforced correctly: "xguest_u:s92:c999999999,c0" passes. -- Petr? From mkosek at redhat.com Wed Sep 26 10:39:08 2012 From: mkosek at redhat.com (Martin Kosek) Date: Wed, 26 Sep 2012 12:39:08 +0200 Subject: [Freeipa-devel] [PATCH] 313 Validate SELinux users in config-mod In-Reply-To: <5062D9BD.2020407@redhat.com> References: <50619B5E.2070805@redhat.com> <5062D828.5000302@redhat.com> <5062D9BD.2020407@redhat.com> Message-ID: <5062DB4C.2010403@redhat.com> On 09/26/2012 12:32 PM, Petr Viktorin wrote: > On 09/26/2012 12:25 PM, Petr Viktorin wrote: >> >> I found strange behavior in validate_selinuxuser. Perhaps it's material >> for another ticket. This command passes validation: >> >> $ ./ipa config_mod >> --ipaselinuxusermapdefault=unconfined_u:s0-s0:c0.c1023 >> --ipaselinuxusermaporder='unconfined_u:s0-s0:c0.c1023$xguest_u:s5-s1:c0-c4.c4,c4:?Why >> >> is stuff allowed here?' >> [...] >> SELinux user map order: >> unconfined_u:s0-s0:c0.c1023$xguest_u:s5-s1:c0-c4.c3.c8,c4:?Why is stuff >> allowed here? >> Default SELinux user: unconfined_u:s0-s0:c0.c1023 >> PAC type: MS-PAC > >> >> Obviously extra info should not be allowed. >> Is "s5-s1" or "c4.c3" valid? Can the first value be higher than the second? >> AFAIK (I'm not an expert though), MCS doesn't allow dashes, so "c0-c4" >> should not be allowed. Chains like "c1.c2.c3" also don't look right. > > > ... Also, the MLS/MCS numeric limits are not enforced correctly: > "xguest_u:s92:c999999999,c0" passes. > > > Right. We can create a ticket to harden the validation if we want. So far, the purpose of this ticket/patch is to make validation of config values consistent with selinuxusermap plugin. I will let Rob to chime in, but I would keep this patch as is. Martin From pspacek at redhat.com Wed Sep 26 10:57:33 2012 From: pspacek at redhat.com (Petr Spacek) Date: Wed, 26 Sep 2012 12:57:33 +0200 Subject: [Freeipa-devel] [PATCH 0070] Fix zone register locking in zr_set_zone_serial_digest() Message-ID: <5062DF9D.1090908@redhat.com> Hello, Fix zone register locking in zr_set_zone_serial_digest(). Zone register has to be locked against simultaneous writes. -- Petr^2 Spacek -------------- next part -------------- A non-text attachment was scrubbed... Name: bind-dyndb-ldap-pspacek-0070-Fix-zone-register-locking-in-zr_set_zone_serial_dige.patch Type: text/x-patch Size: 1148 bytes Desc: not available URL: From pviktori at redhat.com Wed Sep 26 11:38:06 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Wed, 26 Sep 2012 13:38:06 +0200 Subject: [Freeipa-devel] [PATCH] 312 Use custom zonemgr for reverse zones In-Reply-To: <50616E93.80509@redhat.com> References: <50616E93.80509@redhat.com> Message-ID: <5062E91E.5050705@redhat.com> On 09/25/2012 10:42 AM, Martin Kosek wrote: > When DNS is being installed during ipa-{server,dns,replica}-install, > forward and reverse zone is created. However, reverse zone was always > created with default zonemgr even when a custom zonemgr was passed > to the installer as this functionality was missing in function > creating reverse zone. > > Consolidate functions creating forward and reverse zones to avoid > code duplication and errors like this one. Reverse zones are now > created with custom zonemgr (when entered by user). > > https://fedorahosted.org/freeipa/ticket/2790 > ACK. I'm happy to see the duplication go :) -- Petr? From mkosek at redhat.com Wed Sep 26 11:46:19 2012 From: mkosek at redhat.com (Martin Kosek) Date: Wed, 26 Sep 2012 13:46:19 +0200 Subject: [Freeipa-devel] [PATCH] 312 Use custom zonemgr for reverse zones In-Reply-To: <5062E91E.5050705@redhat.com> References: <50616E93.80509@redhat.com> <5062E91E.5050705@redhat.com> Message-ID: <5062EB0B.3080409@redhat.com> On 09/26/2012 01:38 PM, Petr Viktorin wrote: > On 09/25/2012 10:42 AM, Martin Kosek wrote: >> When DNS is being installed during ipa-{server,dns,replica}-install, >> forward and reverse zone is created. However, reverse zone was always >> created with default zonemgr even when a custom zonemgr was passed >> to the installer as this functionality was missing in function >> creating reverse zone. >> >> Consolidate functions creating forward and reverse zones to avoid >> code duplication and errors like this one. Reverse zones are now >> created with custom zonemgr (when entered by user). >> >> https://fedorahosted.org/freeipa/ticket/2790 >> > > ACK. I'm happy to see the duplication go :) > Me too. Pushed to master, ipa-3-0. Martin From tbabej at redhat.com Wed Sep 26 13:05:30 2012 From: tbabej at redhat.com (Tomas Babej) Date: Wed, 26 Sep 2012 15:05:30 +0200 Subject: [Freeipa-devel] [PATCH 0016] Adds port to connection error message in ipa-client-install Message-ID: <5062FD9A.7060802@redhat.com> Hi, Connection error message in ipa-client-install now warns the user about the need of opening 389 port for directory server. https://fedorahosted.org/freeipa/ticket/2816 I think this can be pushed as a one-liner. Tomas -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-tbabej-0016-Adds-port-to-connection-error-message-in-ipa-client-.patch Type: text/x-patch Size: 1391 bytes Desc: not available URL: From tbabej at redhat.com Wed Sep 26 13:23:03 2012 From: tbabej at redhat.com (Tomas Babej) Date: Wed, 26 Sep 2012 15:23:03 +0200 Subject: [Freeipa-devel] [PATCH 0014] Improve user addition to default group in host-add In-Reply-To: <5061894D.9010202@redhat.com> References: <5061894D.9010202@redhat.com> Message-ID: <506301B7.8070107@redhat.com> On 09/25/2012 12:37 PM, Tomas Babej wrote: > Hi, > > On adding new user, host-add tries to make it a member of default > user group. This, however, can raise AlreadyGroupMember when the > user is already member of this group due to automember rule or > default group configured. This patch makes sure AlreadyGroupMember > exception is caught in such cases. > > https://fedorahosted.org/freeipa/ticket/3097 > > Tomas I fixed the typo in the commit message. It refers to the proper command now. Tomas -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-tbabej-0014-2-Improve-user-addition-to-default-group-in-user-add.patch Type: text/x-patch Size: 1657 bytes Desc: not available URL: From atkac at redhat.com Wed Sep 26 14:01:29 2012 From: atkac at redhat.com (Adam Tkac) Date: Wed, 26 Sep 2012 16:01:29 +0200 Subject: [Freeipa-devel] [PATCH 0070] Fix zone register locking in zr_set_zone_serial_digest() In-Reply-To: <5062DF9D.1090908@redhat.com> References: <5062DF9D.1090908@redhat.com> Message-ID: <20120926140129.GA20803@redhat.com> On Wed, Sep 26, 2012 at 12:57:33PM +0200, Petr Spacek wrote: > Hello, > > Fix zone register locking in zr_set_zone_serial_digest(). > > Zone register has to be locked against simultaneous writes. Ack > From ad51025a35efe47542f4379049c8e23d1054726c Mon Sep 17 00:00:00 2001 > From: Petr Spacek > Date: Wed, 26 Sep 2012 12:51:06 +0200 > Subject: [PATCH] Fix zone register locking in zr_set_zone_serial_digest(). > > Zone register has to be locked against simultaneous writes. > > Signed-off-by: Petr Spacek > --- > src/zone_register.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/src/zone_register.c b/src/zone_register.c > index b2b938f3336b23a40d43c85062c9389a2190f3cb..76305730b2e19686568f8a1bc6ac703ed3898fcc 100644 > --- a/src/zone_register.c > +++ b/src/zone_register.c > @@ -370,15 +370,15 @@ zr_set_zone_serial_digest(zone_register_t *zr, dns_name_t *name, > return ISC_R_FAILURE; > } > > - RWLOCK(&zr->rwlock, isc_rwlocktype_read); > + RWLOCK(&zr->rwlock, isc_rwlocktype_write); > > result = dns_rbt_findname(zr->rbt, name, 0, NULL, (void *)&zinfo); > if (result == ISC_R_SUCCESS) { > zinfo->serial = serial; > memcpy(zinfo->digest, digest, RDLIST_DIGESTLENGTH); > } > > - RWUNLOCK(&zr->rwlock, isc_rwlocktype_read); > + RWUNLOCK(&zr->rwlock, isc_rwlocktype_write); > > return result; > } > -- > 1.7.11.4 > -- Adam Tkac, Red Hat, Inc. From mkosek at redhat.com Wed Sep 26 14:12:34 2012 From: mkosek at redhat.com (Martin Kosek) Date: Wed, 26 Sep 2012 16:12:34 +0200 Subject: [Freeipa-devel] [PATCH 0014] Improve user addition to default group in host-add In-Reply-To: <506301B7.8070107@redhat.com> References: <5061894D.9010202@redhat.com> <506301B7.8070107@redhat.com> Message-ID: <50630D52.5090008@redhat.com> On 09/26/2012 03:23 PM, Tomas Babej wrote: > On 09/25/2012 12:37 PM, Tomas Babej wrote: >> Hi, >> >> On adding new user, host-add tries to make it a member of default >> user group. This, however, can raise AlreadyGroupMember when the >> user is already member of this group due to automember rule or >> default group configured. This patch makes sure AlreadyGroupMember >> exception is caught in such cases. >> >> https://fedorahosted.org/freeipa/ticket/3097 >> >> Tomas > > I fixed the typo in the commit message. It refers to the proper command now. > > Tomas I would also like to see the tests that Petr Viktorin already asked for. Setting an automember default group to ipausers, adding a user and checking the result should be enough. Martin From pspacek at redhat.com Wed Sep 26 14:55:27 2012 From: pspacek at redhat.com (Petr Spacek) Date: Wed, 26 Sep 2012 16:55:27 +0200 Subject: [Freeipa-devel] [PATCH 0070] Fix zone register locking in zr_set_zone_serial_digest() In-Reply-To: <20120926140129.GA20803@redhat.com> References: <5062DF9D.1090908@redhat.com> <20120926140129.GA20803@redhat.com> Message-ID: <5063175F.6080301@redhat.com> On 09/26/2012 04:01 PM, Adam Tkac wrote: > On Wed, Sep 26, 2012 at 12:57:33PM +0200, Petr Spacek wrote: >> Hello, >> >> Fix zone register locking in zr_set_zone_serial_digest(). >> >> Zone register has to be locked against simultaneous writes. > > Ack Pushed to master: 09bdbfc807a63c19d171af5fe3b13370483f26ea Petr^2 Spacek From mkosek at redhat.com Wed Sep 26 15:44:53 2012 From: mkosek at redhat.com (Martin Kosek) Date: Wed, 26 Sep 2012 17:44:53 +0200 Subject: [Freeipa-devel] [PATCH 0015] Restrict admins group modifications In-Reply-To: <5061AAAB.7020801@redhat.com> References: <5061A202.2090203@redhat.com> <5061A432.3090709@redhat.com> <5061AAAB.7020801@redhat.com> Message-ID: <506322F5.4000901@redhat.com> On 09/25/2012 02:59 PM, Tomas Babej wrote: > On 09/25/2012 02:31 PM, Martin Kosek wrote: >> On 09/25/2012 02:22 PM, Tomas Babej wrote: >>> Hi, >>> >>> Group-mod command no longer allows --rename and/or --external >>> changes made to the admins group. In such cases, ProtectedEntryError >>> is being raised. >>> >>> https://fedorahosted.org/freeipa/ticket/3098 >>> >>> Tomas >>> >> Just based on a quick glance, I see few issues: >> >> 1) I would like to see unit tests for this scenario >> >> 2) Some overlooked debug output: >> >> + self.log.info(str(keys)) >> >> Martin > I removed the unfortunate debug output and added two unit tests to test the > scenarios. > > Tomas I finally got to a proper review and here it is: 1) I think we should use some common global variable containing protected groups and not define it in every command separately (duplication -> errors). One is already used: ... protected_group_name = u'admins' ... I would like to see that to be used in both group-del and group-mod. I also think we should protect "trust admins" too as this group is also encoded in trust ACI, i.e. using a global variable like this one: PROTECTED_GROUPS = (u'admins', u'trust admins') 2) Minor issue, I think instead of: + is_admins_group = u'admins' in keys a more common pattern in IPA is the following: + is_admins_group = keys[-1] == u'admins' 3) We usually do not end our error messages in exceptions with ".": ... + raise errors.ProtectedEntryError(label=u'group', key=u'admins', reason=u'Cannot be renamed.') ... + raise errors.ProtectedEntryError(label=u'group', key=u'admins', reason=u'Cannot support external non-IPA members.') ... Otherwise the patch looks OK. Martin From rcritten at redhat.com Wed Sep 26 18:20:45 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 26 Sep 2012 14:20:45 -0400 Subject: [Freeipa-devel] Announcing FreeIPA v3.0.0 release candidate 1 Message-ID: <5063477D.8010003@redhat.com> The FreeIPA team is proud to announce version FreeIPA v3.0.0 rc 1. It can be downloaded from http://www.freeipa.org/page/Downloads. A build is available in the Fedora 18 and rawhide repositories or for Fedora 17 via the freeipa-devel repo on www.freeipa.org: http://freeipa.org/downloads/freeipa-devel.repo . To install in Fedora 17 the updates repo repository needs to be enabled as well. For Fedora 17 you will also need libldb-1.1.12-1 installed for sssd to work. There are no dependencies on this package. For additional information see the AD Trust design page http://freeipa.org/page/IPAv3_AD_trust and the AD Trust testing page http://freeipa.org/page/IPAv3_testing_AD_trust. == Highlights since 3.0.0 beta 3 == * Support for the Dogtag CA version 10 * Verification when setting up AD trust * New ipa-client-install option to disable OpenSSH client configuration. * Expand Referential Integrity checks on hosts, SUDO and HBAC rule referential attributes * Run the CLEANALLRUV task when deleting a replication agreement to remove replication meta-data about removed master. See the ipa-replica-manage man page for the list of new commands related to CLEANALLRUV command. * Try to prevent orphaning other servers when deleting a master. == Upgrading == An IPA server can be upgraded simply by installing updated rpms. The server does not need to be shut down in advance. Please note, that the referential integrity extension requires an extended set of indexes to be configured. RPM update for an IPA server with a excessive number of hosts, SUDO or HBAC entries may require several minutes to finish. If you have multiple servers you may upgrade them one at a time. It is expected that all servers will be upgraded in a relatively short period (days or weeks not months). They should be able to co-exist peacefully but new features will not be available on old servers and enrolling a new client against an old server will result in the SSH keys not being uploaded. Downgrading a server once upgraded is not supported. Upgrading from 2.2.0 should work but has not been fully tested. Proceed with caution. An enrolled client does not need the new packages installed unless you want to re-enroll it. SSH keys for already installed clients are not uploaded, you will have to re-enroll the client or manually upload the keys. == Feedback == Please provide comments, bugs and other feedback via the freeipa-devel mailing list: http://www.redhat.com/mailman/listinfo/freeipa-devel == Detailed changelog == Ade Lee (1): * Modifications to install scripts for dogtag 10 Alexander Bokovoy (5): * Add verification of the AD trust * validate SID for trusted domain when adding/modifying ID range * Fix error messages and use proper ImportError for dcerpc import * Add documentation for 'ipa trust' set of commands * Document use of external group membership Jan Cholasta (3): * Add the SSH service to SSSD config file before trying to activate it. * Add --no-ssh option to ipa-client-install to disable OpenSSH client configuration. * SSHPublicKey.fingerprint_dns_sha1 should return unicode value. Martin Kosek (8): * Fix addattr internal error * Add attributeTypes to safe schema updater * Amend memberAllowCmd and memberDenyCmd attribute types * Run index task in ldap updater only when needed * Expand Referential Integrity checks * Properly convert DN in ipa-client-install * Use default reverse zone consistently * Fix idrange plugin help Petr Viktorin (7): * ipa-client-install: Obtain host TGT from one specific KDC * Fix server installation * Use temporary key cache for host key in server installation * Update the pot file (translation source) * Use Dogtag 10 only when it is available * Only stop the main DS instance when upgrading it * Use correct Dogtag port in ipaserver.install.certs Petr Vobornik (4): * Prevent opening of multiple dirty dialogs on navigation * JSON serialization of long type * Show trust status in add success notification * Fix integer validation when boundary value is empty string Rob Crittenden (3): * Set SELinux default context to unconfined_u:s0-s0:c0.c1023 * Run the CLEANALLRUV task when deleting a replication agreement. * When deleting a master, try to prevent orphaning other servers. Sumit Bose (3): * ipasam: Fixes build with samba4 rc1 * Set master_kdc and dns_lookup_kdc to true * Update krb5.conf during ipa-adtrust-install Tomas Babej (2): * Make sure selinuxusemap behaves consistently to HBAC rule * Improves sssd.conf handling during ipa-client uninstall Yuri Chornoivan (1): * Fix various typos. From rcritten at redhat.com Wed Sep 26 18:31:03 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 26 Sep 2012 14:31:03 -0400 Subject: [Freeipa-devel] [PATCH] 313 Validate SELinux users in config-mod In-Reply-To: <5062DB4C.2010403@redhat.com> References: <50619B5E.2070805@redhat.com> <5062D828.5000302@redhat.com> <5062D9BD.2020407@redhat.com> <5062DB4C.2010403@redhat.com> Message-ID: <506349E7.2030008@redhat.com> Martin Kosek wrote: > On 09/26/2012 12:32 PM, Petr Viktorin wrote: >> On 09/26/2012 12:25 PM, Petr Viktorin wrote: >>> >>> I found strange behavior in validate_selinuxuser. Perhaps it's material >>> for another ticket. This command passes validation: >>> >>> $ ./ipa config_mod >>> --ipaselinuxusermapdefault=unconfined_u:s0-s0:c0.c1023 >>> --ipaselinuxusermaporder='unconfined_u:s0-s0:c0.c1023$xguest_u:s5-s1:c0-c4.c4,c4:?Why >>> >>> is stuff allowed here?' >>> [...] >>> SELinux user map order: >>> unconfined_u:s0-s0:c0.c1023$xguest_u:s5-s1:c0-c4.c3.c8,c4:?Why is stuff >>> allowed here? >>> Default SELinux user: unconfined_u:s0-s0:c0.c1023 >>> PAC type: MS-PAC >> >>> >>> Obviously extra info should not be allowed. >>> Is "s5-s1" or "c4.c3" valid? Can the first value be higher than the second? >>> AFAIK (I'm not an expert though), MCS doesn't allow dashes, so "c0-c4" >>> should not be allowed. Chains like "c1.c2.c3" also don't look right. >> >> >> ... Also, the MLS/MCS numeric limits are not enforced correctly: >> "xguest_u:s92:c999999999,c0" passes. >> >> >> > > Right. We can create a ticket to harden the validation if we want. So far, the > purpose of this ticket/patch is to make validation of config values consistent > with selinuxusermap plugin. I will let Rob to chime in, but I would keep this > patch as is. > > Martin > Yes, please open another ticket for the validation issues. rob From abokovoy at redhat.com Wed Sep 26 18:43:20 2012 From: abokovoy at redhat.com (Alexander Bokovoy) Date: Wed, 26 Sep 2012 14:43:20 -0400 (EDT) Subject: [Freeipa-devel] Announcing FreeIPA v3.0.0 release candidate 1 In-Reply-To: <5063477D.8010003@redhat.com> Message-ID: <1817590715.2490789.1348685000903.JavaMail.root@redhat.com> Hi, small addition -- in order to get trusts created properly on new installs you'll need to apply my patch 0080 https://www.redhat.com/archives/freeipa-devel/2012-September/msg00426.html this is one fix that we missed in RC1. :( -- / Alexander Bokovoy ----- Original Message ----- > From: "Rob Crittenden" > To: freeipa-interest at redhat.com, freeipa-users-owner at redhat.com, "freeipa-devel" > Sent: Wednesday, September 26, 2012 9:20:45 PM > Subject: [Freeipa-devel] Announcing FreeIPA v3.0.0 release candidate 1 > > The FreeIPA team is proud to announce version FreeIPA v3.0.0 rc 1. > > It can be downloaded from http://www.freeipa.org/page/Downloads. > > A build is available in the Fedora 18 and rawhide repositories or for > Fedora 17 via the freeipa-devel repo on www.freeipa.org: > http://freeipa.org/downloads/freeipa-devel.repo . To install in > Fedora > 17 the updates repo repository needs to be enabled as well. For > Fedora > 17 you will also need libldb-1.1.12-1 installed for sssd to work. > There > are no dependencies on this package. > > For additional information see the AD Trust design page > http://freeipa.org/page/IPAv3_AD_trust and the AD Trust testing page > http://freeipa.org/page/IPAv3_testing_AD_trust. > > == Highlights since 3.0.0 beta 3 == > > * Support for the Dogtag CA version 10 > * Verification when setting up AD trust > * New ipa-client-install option to disable OpenSSH client > configuration. > * Expand Referential Integrity checks on hosts, SUDO and HBAC rule > referential attributes > * Run the CLEANALLRUV task when deleting a replication agreement to > remove replication meta-data about removed master. See the > ipa-replica-manage man page for the list of new commands related to > CLEANALLRUV command. > * Try to prevent orphaning other servers when deleting a master. > > == Upgrading == > > An IPA server can be upgraded simply by installing updated rpms. The > server does not need to be shut down in advance. > > Please note, that the referential integrity extension requires an > extended set of indexes to be configured. RPM update for an IPA > server > with a excessive number of hosts, SUDO or HBAC entries may require > several minutes to finish. > > If you have multiple servers you may upgrade them one at a time. It > is > expected that all servers will be upgraded in a relatively short > period > (days or weeks not months). They should be able to co-exist > peacefully > but new features will not be available on old servers and enrolling a > new client against an old server will result in the SSH keys not > being > uploaded. > > Downgrading a server once upgraded is not supported. > > Upgrading from 2.2.0 should work but has not been fully tested. > Proceed > with caution. > > An enrolled client does not need the new packages installed unless > you > want to re-enroll it. SSH keys for already installed clients are not > uploaded, you will have to re-enroll the client or manually upload > the keys. > > == Feedback == > > Please provide comments, bugs and other feedback via the > freeipa-devel > mailing list: http://www.redhat.com/mailman/listinfo/freeipa-devel > > == Detailed changelog == > > Ade Lee (1): > * Modifications to install scripts for dogtag 10 > > Alexander Bokovoy (5): > * Add verification of the AD trust > * validate SID for trusted domain when adding/modifying ID range > * Fix error messages and use proper ImportError for dcerpc import > * Add documentation for 'ipa trust' set of commands > * Document use of external group membership > > Jan Cholasta (3): > * Add the SSH service to SSSD config file before trying to activate > it. > * Add --no-ssh option to ipa-client-install to disable OpenSSH client > configuration. > * SSHPublicKey.fingerprint_dns_sha1 should return unicode value. > > Martin Kosek (8): > * Fix addattr internal error > * Add attributeTypes to safe schema updater > * Amend memberAllowCmd and memberDenyCmd attribute types > * Run index task in ldap updater only when needed > * Expand Referential Integrity checks > * Properly convert DN in ipa-client-install > * Use default reverse zone consistently > * Fix idrange plugin help > > Petr Viktorin (7): > * ipa-client-install: Obtain host TGT from one specific KDC > * Fix server installation > * Use temporary key cache for host key in server installation > * Update the pot file (translation source) > * Use Dogtag 10 only when it is available > * Only stop the main DS instance when upgrading it > * Use correct Dogtag port in ipaserver.install.certs > > Petr Vobornik (4): > * Prevent opening of multiple dirty dialogs on navigation > * JSON serialization of long type > * Show trust status in add success notification > * Fix integer validation when boundary value is empty string > > Rob Crittenden (3): > * Set SELinux default context to unconfined_u:s0-s0:c0.c1023 > * Run the CLEANALLRUV task when deleting a replication agreement. > * When deleting a master, try to prevent orphaning other servers. > > Sumit Bose (3): > * ipasam: Fixes build with samba4 rc1 > * Set master_kdc and dns_lookup_kdc to true > * Update krb5.conf during ipa-adtrust-install > > Tomas Babej (2): > * Make sure selinuxusemap behaves consistently to HBAC rule > * Improves sssd.conf handling during ipa-client uninstall > > Yuri Chornoivan (1): > * Fix various typos. > > _______________________________________________ > Freeipa-devel mailing list > Freeipa-devel at redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-devel > From rcritten at redhat.com Wed Sep 26 18:58:25 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 26 Sep 2012 14:58:25 -0400 Subject: [Freeipa-devel] [PATCH] 314-315 Limit unindexed searches In-Reply-To: <5061D833.10204@redhat.com> References: <5061D833.10204@redhat.com> Message-ID: <50635051.3060809@redhat.com> Martin Kosek wrote: > These 2 patches significantly limit the number of unindexed LDAP searches we do > in IPA. I used our unit test suite as a good source of different LDAP searches > run by our command suite. > > Most of the remaining unindexed searches are produced either by our general > term search ("ipa service-find TERM") which hit every object parameter and DNS > commands (idnsname is not indexed yet). I am thinking about indexing about > idnsName as well... > Patch 314 looks ok. It addresses ticket 3025 as well (ipakrbprincipalalias). For 315 I'd prefer we add a new exception type to mirror ldap.NOT_ALLOWED_ON_NONLEAF and catch that in delete instead of DatabaseError. rob From rcritten at redhat.com Wed Sep 26 19:32:23 2012 From: rcritten at redhat.com (Rob Crittenden) Date: Wed, 26 Sep 2012 15:32:23 -0400 Subject: [Freeipa-devel] [PATCH 0016] Adds port to connection error message in ipa-client-install In-Reply-To: <5062FD9A.7060802@redhat.com> References: <5062FD9A.7060802@redhat.com> Message-ID: <50635847.5070602@redhat.com> Tomas Babej wrote: > Hi, > > Connection error message in ipa-client-install now warns the user > about the need of opening 389 port for directory server. > > https://fedorahosted.org/freeipa/ticket/2816 > > I think this can be pushed as a one-liner. I think we should list all ports that are required for client enrollment. From my calculations we need at a minimum tcp ports 80 and 389, either or both udp/tcp for port 88 and if NTP is enabled 123 udp for enrollment alone. The NTP failure won't cause enrollment to fail though, so we may be able to skip that. Similarly 464 should be enabled but we don't use it during enrollment. rob From ssorce at redhat.com Wed Sep 26 22:36:40 2012 From: ssorce at redhat.com (Simo Sorce) Date: Wed, 26 Sep 2012 18:36:40 -0400 (EDT) Subject: [Freeipa-devel] [PATCH] support AES for cross-realm TGTs In-Reply-To: <26889845.2398987.1348698998522.JavaMail.root@redhat.com> Message-ID: <1557234042.2399022.1348699000344.JavaMail.root@redhat.com> This patch allows Windows to send us TGTs using AES. Simo. -- Simo Sorce * Red Hat, Inc. * New York -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Add-support-for-using-AES-fo-cross-realm-TGTs.patch Type: text/x-patch Size: 1633 bytes Desc: not available URL: From mkosek at redhat.com Thu Sep 27 07:53:08 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 27 Sep 2012 09:53:08 +0200 Subject: [Freeipa-devel] [PATCH] 314-315 Limit unindexed searches In-Reply-To: <50635051.3060809@redhat.com> References: <5061D833.10204@redhat.com> <50635051.3060809@redhat.com> Message-ID: <506405E4.3060400@redhat.com> On 09/26/2012 08:58 PM, Rob Crittenden wrote: > Martin Kosek wrote: >> These 2 patches significantly limit the number of unindexed LDAP searches we do >> in IPA. I used our unit test suite as a good source of different LDAP searches >> run by our command suite. >> >> Most of the remaining unindexed searches are produced either by our general >> term search ("ipa service-find TERM") which hit every object parameter and DNS >> commands (idnsname is not indexed yet). I am thinking about indexing about >> idnsName as well... >> > > Patch 314 looks ok. It addresses ticket 3025 as well (ipakrbprincipalalias). > I added this ticket number to the patch too. Btw. managedBy attribute mentioned in the ticket was already added to indexed in ticket #2866. > For 315 I'd prefer we add a new exception type to mirror > ldap.NOT_ALLOWED_ON_NONLEAF and catch that in delete instead of DatabaseError. This is a very good catch, it will make this procedure safer when there is an actual DatabaseError raised. Updated patches attached. Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-314-2-add-indexes.patch Type: text/x-patch Size: 2371 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-315-2-unindexed-searche-on-del.patch Type: text/x-patch Size: 3265 bytes Desc: not available URL: From mkosek at redhat.com Thu Sep 27 07:59:46 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 27 Sep 2012 09:59:46 +0200 Subject: [Freeipa-devel] [PATCH] 313 Validate SELinux users in config-mod In-Reply-To: <506349E7.2030008@redhat.com> References: <50619B5E.2070805@redhat.com> <5062D828.5000302@redhat.com> <5062D9BD.2020407@redhat.com> <5062DB4C.2010403@redhat.com> <506349E7.2030008@redhat.com> Message-ID: <50640772.4070800@redhat.com> On 09/26/2012 08:31 PM, Rob Crittenden wrote: > Martin Kosek wrote: >> On 09/26/2012 12:32 PM, Petr Viktorin wrote: >>> On 09/26/2012 12:25 PM, Petr Viktorin wrote: >>>> >>>> I found strange behavior in validate_selinuxuser. Perhaps it's material >>>> for another ticket. This command passes validation: >>>> >>>> $ ./ipa config_mod >>>> --ipaselinuxusermapdefault=unconfined_u:s0-s0:c0.c1023 >>>> --ipaselinuxusermaporder='unconfined_u:s0-s0:c0.c1023$xguest_u:s5-s1:c0-c4.c4,c4:?Why >>>> >>>> >>>> is stuff allowed here?' >>>> [...] >>>> SELinux user map order: >>>> unconfined_u:s0-s0:c0.c1023$xguest_u:s5-s1:c0-c4.c3.c8,c4:?Why is stuff >>>> allowed here? >>>> Default SELinux user: unconfined_u:s0-s0:c0.c1023 >>>> PAC type: MS-PAC >>> >>>> >>>> Obviously extra info should not be allowed. >>>> Is "s5-s1" or "c4.c3" valid? Can the first value be higher than the second? >>>> AFAIK (I'm not an expert though), MCS doesn't allow dashes, so "c0-c4" >>>> should not be allowed. Chains like "c1.c2.c3" also don't look right. >>> >>> >>> ... Also, the MLS/MCS numeric limits are not enforced correctly: >>> "xguest_u:s92:c999999999,c0" passes. >>> >>> >>> >> >> Right. We can create a ticket to harden the validation if we want. So far, the >> purpose of this ticket/patch is to make validation of config values consistent >> with selinuxusermap plugin. I will let Rob to chime in, but I would keep this >> patch as is. >> >> Martin >> > > Yes, please open another ticket for the validation issues. > > rob https://fedorahosted.org/freeipa/ticket/3119 Martin From pviktori at redhat.com Thu Sep 27 08:26:51 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Thu, 27 Sep 2012 10:26:51 +0200 Subject: [Freeipa-devel] [PATCH] Changes to use a single database for dogtag and IPA In-Reply-To: <1348113533.2575.14.camel@aleeredhat.laptop> References: <1348113533.2575.14.camel@aleeredhat.laptop> Message-ID: <50640DCB.2060901@redhat.com> On 09/20/2012 05:58 AM, Ade Lee wrote: > Changes to use a single database for dogtag and IPA > > New servers that are installed with dogtag 10 instances will use > a single database instance for dogtag and IPA, albeit with different > suffixes. Dogtag will communicate with the instance through a > database user with permissions to modify the dogtag suffix only. > This user will authenticate using client auth using the subsystem cert > for the instance. > > This patch includes changes to allow the creation of masters and clones > with single ds instances. > > I have tested being able to create a master and a clone using f17 and > dogtag 10. Note that you will need to use the latest builds on the > dogtag repo to get some changes that were checked in today. We'll kick > off another official f18 dogtag build in a day or so. > > This is a pretty big change - so I expect many issues to come up as > things get tested. But as this will take awhile to get resolved, its > better to get this out for review as fast as possible. > > Happy reviewing. > > Ade > > Attaching a rebased patch with a couple of style issues fixed. - PEP8 compliance (remove trailing whitespace, use parentheses rather than \ for line continuation, wrap touched lines at 80 characters) - for files, use the with statement instead of the "open/close sandwich" - don't mix tabs and spaces in install/share/certmap.conf.template I've also adjusted the spec file, as we need dogtag 10.0 and pki-server now obsoletes pki-setup. I still need selinux in permissive mode to install on f17, and I still need to exclude *.i686 packages when updating. -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-alee-0001-01-Changes-to-use-a-single-database-for-dogtag-and-IPA.patch Type: text/x-patch Size: 35382 bytes Desc: not available URL: From pviktori at redhat.com Thu Sep 27 08:42:11 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Thu, 27 Sep 2012 10:42:11 +0200 Subject: [Freeipa-devel] [PATCH] 313 Validate SELinux users in config-mod In-Reply-To: <50640772.4070800@redhat.com> References: <50619B5E.2070805@redhat.com> <5062D828.5000302@redhat.com> <5062D9BD.2020407@redhat.com> <5062DB4C.2010403@redhat.com> <506349E7.2030008@redhat.com> <50640772.4070800@redhat.com> Message-ID: <50641163.1010701@redhat.com> On 09/27/2012 09:59 AM, Martin Kosek wrote: > On 09/26/2012 08:31 PM, Rob Crittenden wrote: >> Martin Kosek wrote: >>> On 09/26/2012 12:32 PM, Petr Viktorin wrote: >>>> On 09/26/2012 12:25 PM, Petr Viktorin wrote: >>>>> >>>>> I found strange behavior in validate_selinuxuser. Perhaps it's material >>>>> for another ticket. This command passes validation: >>>>> >>>>> $ ./ipa config_mod >>>>> --ipaselinuxusermapdefault=unconfined_u:s0-s0:c0.c1023 >>>>> --ipaselinuxusermaporder='unconfined_u:s0-s0:c0.c1023$xguest_u:s5-s1:c0-c4.c4,c4:?Why >>>>> >>>>> >>>>> is stuff allowed here?' >>>>> [...] >>>>> SELinux user map order: >>>>> unconfined_u:s0-s0:c0.c1023$xguest_u:s5-s1:c0-c4.c3.c8,c4:?Why is stuff >>>>> allowed here? >>>>> Default SELinux user: unconfined_u:s0-s0:c0.c1023 >>>>> PAC type: MS-PAC >>>> >>>>> >>>>> Obviously extra info should not be allowed. >>>>> Is "s5-s1" or "c4.c3" valid? Can the first value be higher than the second? >>>>> AFAIK (I'm not an expert though), MCS doesn't allow dashes, so "c0-c4" >>>>> should not be allowed. Chains like "c1.c2.c3" also don't look right. >>>> >>>> >>>> ... Also, the MLS/MCS numeric limits are not enforced correctly: >>>> "xguest_u:s92:c999999999,c0" passes. >>>> >>>> >>>> >>> >>> Right. We can create a ticket to harden the validation if we want. So far, the >>> purpose of this ticket/patch is to make validation of config values consistent >>> with selinuxusermap plugin. I will let Rob to chime in, but I would keep this >>> patch as is. >>> >>> Martin >>> >> >> Yes, please open another ticket for the validation issues. >> >> rob > > https://fedorahosted.org/freeipa/ticket/3119 > > Martin > ACK for the patch -- Petr? From mkosek at redhat.com Thu Sep 27 08:44:36 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 27 Sep 2012 10:44:36 +0200 Subject: [Freeipa-devel] [PATCH] 313 Validate SELinux users in config-mod In-Reply-To: <50641163.1010701@redhat.com> References: <50619B5E.2070805@redhat.com> <5062D828.5000302@redhat.com> <5062D9BD.2020407@redhat.com> <5062DB4C.2010403@redhat.com> <506349E7.2030008@redhat.com> <50640772.4070800@redhat.com> <50641163.1010701@redhat.com> Message-ID: <506411F4.60008@redhat.com> On 09/27/2012 10:42 AM, Petr Viktorin wrote: > On 09/27/2012 09:59 AM, Martin Kosek wrote: >> On 09/26/2012 08:31 PM, Rob Crittenden wrote: >>> Martin Kosek wrote: >>>> On 09/26/2012 12:32 PM, Petr Viktorin wrote: >>>>> On 09/26/2012 12:25 PM, Petr Viktorin wrote: >>>>>> >>>>>> I found strange behavior in validate_selinuxuser. Perhaps it's material >>>>>> for another ticket. This command passes validation: >>>>>> >>>>>> $ ./ipa config_mod >>>>>> --ipaselinuxusermapdefault=unconfined_u:s0-s0:c0.c1023 >>>>>> --ipaselinuxusermaporder='unconfined_u:s0-s0:c0.c1023$xguest_u:s5-s1:c0-c4.c4,c4:?Why >>>>>> >>>>>> >>>>>> >>>>>> is stuff allowed here?' >>>>>> [...] >>>>>> SELinux user map order: >>>>>> unconfined_u:s0-s0:c0.c1023$xguest_u:s5-s1:c0-c4.c3.c8,c4:?Why is stuff >>>>>> allowed here? >>>>>> Default SELinux user: unconfined_u:s0-s0:c0.c1023 >>>>>> PAC type: MS-PAC >>>>> >>>>>> >>>>>> Obviously extra info should not be allowed. >>>>>> Is "s5-s1" or "c4.c3" valid? Can the first value be higher than the second? >>>>>> AFAIK (I'm not an expert though), MCS doesn't allow dashes, so "c0-c4" >>>>>> should not be allowed. Chains like "c1.c2.c3" also don't look right. >>>>> >>>>> >>>>> ... Also, the MLS/MCS numeric limits are not enforced correctly: >>>>> "xguest_u:s92:c999999999,c0" passes. >>>>> >>>>> >>>>> >>>> >>>> Right. We can create a ticket to harden the validation if we want. So far, the >>>> purpose of this ticket/patch is to make validation of config values consistent >>>> with selinuxusermap plugin. I will let Rob to chime in, but I would keep this >>>> patch as is. >>>> >>>> Martin >>>> >>> >>> Yes, please open another ticket for the validation issues. >>> >>> rob >> >> https://fedorahosted.org/freeipa/ticket/3119 >> >> Martin >> > > ACK for the patch > Pushed to master, ipa-3-0. Martin From mkosek at redhat.com Thu Sep 27 11:35:41 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 27 Sep 2012 13:35:41 +0200 Subject: [Freeipa-devel] [PATCH] 316 Improve DN usage in ipa-client-install Message-ID: <50643A0D.3020603@redhat.com> A hotfix pushed in a scope of ticket 3088 forced conversion of DN object (baseDN) in IPA client discovery so that ipa-client-install does not crash when creating an IPA default.conf. Since this is not a preferred way to handle DN objects, improve its usage: - make sure, that baseDN retrieved by client discovery is always a DN object - update ipachangeconf.py code to handle strings better and instead of concatenating objects, make sure they are converted to string first As a side-effect of ipachangeconf changes, default.conf config file generated by ipa-client-install has no longer empty new line at the end of a file. https://fedorahosted.org/freeipa/ticket/3088 -- Martin Kosek Senior Software Engineer - Identity Management Team Red Hat Inc. -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-316-improve-dn-usage-in-ipa-client-install.patch Type: text/x-patch Size: 11147 bytes Desc: not available URL: From sbose at redhat.com Thu Sep 27 11:57:24 2012 From: sbose at redhat.com (Sumit Bose) Date: Thu, 27 Sep 2012 13:57:24 +0200 Subject: [Freeipa-devel] [PATCH] 0080 rewrite SID comparison to take into account different SID forms In-Reply-To: <20120925144057.GM17454@redhat.com> References: <20120925144057.GM17454@redhat.com> Message-ID: <20120927115724.GF6816@localhost.localdomain> On Tue, Sep 25, 2012 at 05:40:57PM +0300, Alexander Bokovoy wrote: > Hi, > > Domain validator code in ipaserver/dcerpc.py verifies that a SID belongs > to one of our trusted domains. This verification was expecting that SID > is for some resource within trusted domain and ignored the case when it > is the SID of the trusted domain, i.e. when sid has form like > S-1-5-21-16904141-148189700-2149043814 rather than > S-1-5-21-16904141-148189700-2149043814-512 (Domain Admins). > > The latter is what idrange-add command uses. > > So comparing SID with SID was done by stripping last component (RID). > In case of idrange-add stripping last RID was making a SID that could > never compare to a trusted domain SID. > > Somehow the code worked for me in Fedora and started failing on RHEL6. > > -- > / Alexander Bokovoy ACK bye, Sumit From mkosek at redhat.com Thu Sep 27 12:01:11 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 27 Sep 2012 14:01:11 +0200 Subject: [Freeipa-devel] [PATCH] 0080 rewrite SID comparison to take into account different SID forms In-Reply-To: <20120927115724.GF6816@localhost.localdomain> References: <20120925144057.GM17454@redhat.com> <20120927115724.GF6816@localhost.localdomain> Message-ID: <50644007.8030007@redhat.com> On 09/27/2012 01:57 PM, Sumit Bose wrote: > On Tue, Sep 25, 2012 at 05:40:57PM +0300, Alexander Bokovoy wrote: >> Hi, >> >> Domain validator code in ipaserver/dcerpc.py verifies that a SID belongs >> to one of our trusted domains. This verification was expecting that SID >> is for some resource within trusted domain and ignored the case when it >> is the SID of the trusted domain, i.e. when sid has form like >> S-1-5-21-16904141-148189700-2149043814 rather than >> S-1-5-21-16904141-148189700-2149043814-512 (Domain Admins). >> >> The latter is what idrange-add command uses. >> >> So comparing SID with SID was done by stripping last component (RID). >> In case of idrange-add stripping last RID was making a SID that could >> never compare to a trusted domain SID. >> >> Somehow the code worked for me in Fedora and started failing on RHEL6. >> >> -- >> / Alexander Bokovoy > > ACK > > bye, > Sumit > Pushed to master, ipa-3-0. Martin From pviktori at redhat.com Thu Sep 27 12:15:19 2012 From: pviktori at redhat.com (Petr Viktorin) Date: Thu, 27 Sep 2012 14:15:19 +0200 Subject: [Freeipa-devel] [PATCH] 0084 Fix NS records in installation Message-ID: <50644357.4090903@redhat.com> This fixes a regression in the recent DNS zone consolidation work. -- Petr? -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-pviktori-0084-Fix-NS-records-in-installation.patch Type: text/x-patch Size: 1844 bytes Desc: not available URL: From mkosek at redhat.com Thu Sep 27 12:28:58 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 27 Sep 2012 14:28:58 +0200 Subject: [Freeipa-devel] [PATCH] 317 Improve StrEnum validation error message Message-ID: <5064468A.5000204@redhat.com> Do not print list of possible values as "%r" but simply as a list of quoted values which should make it easier to read for users. Also add a special case when there is just one allowed value. https://fedorahosted.org/freeipa/ticket/2869 Examples of the improved Enum validation error messages: # ipa automember-add foo --type=bar ipa: ERROR: invalid 'type': must be one of 'group', 'hostgroup' # ipa trust-add foo --type=foo ipa: ERROR: invalid 'type': must be 'ad' Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: freeipa-mkosek-317-improve-strenum-validation-error-message.patch Type: text/x-patch Size: 3336 bytes Desc: not available URL: From mkosek at redhat.com Thu Sep 27 13:24:57 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 27 Sep 2012 15:24:57 +0200 Subject: [Freeipa-devel] [PATCH] 0085 Don't use bare except: clauses in ipa-client-install In-Reply-To: <5061D4DF.90207@redhat.com> References: <5061D4DF.90207@redhat.com> Message-ID: <506453A9.2050307@redhat.com> On 09/25/2012 05:59 PM, Petr Viktorin wrote: > https://fedorahosted.org/freeipa/ticket/2941 > > Simo, do you remember how you configured resolv.conf and when you pressed > Ctrl+C? There's a lot of code called from ipa-replica-install that could cause > the bug. > > Here I went for the most obvious places where we ignore KeyboardInterrupt. I > will look for others. > > I'll also do a more thorough audit of the installer code when I work on > https://fedorahosted.org/freeipa/ticket/2652. > > I did not reproduce the issue either, but the changes you made are OK and should improve the situation at minimum. The issue may have been fixed during 3.0 development as a side-effect of other patch. ACK. Pushed to master, ipa-3-0. Martin From mkosek at redhat.com Thu Sep 27 14:01:54 2012 From: mkosek at redhat.com (Martin Kosek) Date: Thu, 27 Sep 2012 16:01:54 +0200 Subject: [Freeipa-devel] [PATCH] 0084 Fix NS records in installation In-Reply-To: <50644357.4090903@redhat.com> References: <50644357.4090903@redhat.com> Message-ID: <50645C52.7000200@redhat.com> On 09/27/2012 02:15 PM, Petr Viktorin wrote: > This fixes a regression in the recent DNS zone consolidation work. > ACK. Pushed to master, ipa-3-0. Martin From sbose at redhat.com Thu Sep 27 20:21:31 2012 From: sbose at redhat.com (Sumit Bose) Date: Thu, 27 Sep 2012 22:21:31 +0200 Subject: [Freeipa-devel] [PATCH] Improve performance of get_group_sids() In-Reply-To: <20120710210420.GD922@localhost.localdomain> References: <20120710210420.GD922@localhost.localdomain> Message-ID: <20120927202131.GI6816@localhost.localdomain> On Tue, Jul 10, 2012 at 11:04:20PM +0200, Sumit Bose wrote: > Hi, > > the following two patches are the first step to fix > https://fedorahosted.org/freeipa/ticket/2881. Unit tests with time > measurements are added and the performance of the get_group_sids() > function is improved by an order of magnitude. > > The caching of the LDAP results is still missing. I will finish this > after my PTO. But I haven't started to work on this. So if you think it > should be fixed earlier feel free to take the ticket. > > bye, > Sumit > For some reasons I never got Simo's review mail and I just recently > discovered it in the archive. His comments were: > > Nack, for minor issues. > > Patch1: > - Please create a tests/ directory in daemons/ipa-kdb for the tests done > > - Please fit the whole file within 79 columns, there are a number of > lines towards the end of the tests that go beyond that. > done (I hope I haven't missed too many) > - There is a spurious comment at the top: > +/* talloc leak checks written by Martin Nagy for sssd */ removed > > - please add a dlinkslist.h file to /util instead of copying macros in > the tests file added together with ipa-kdb/tests/common_check.[ch] > > Patch2: > > - Needs spacing in arithmetic operations. This line: > + idx = group_sids_buf+(p*sid_len); > should be: > + idx = group_sids_buf + (p * sid_len); > and other similar cases. done > > - Should we align the sids in the big memory allocation you make ? > + dom_sid_str_len = strlen(dom_sid_str); > + sid_len = dom_sid_str_len + 12; > this can result in any alignment, should we align32/64 sid_len ? > It would waste a bit of space, but may be beneficial on some archs. done, I took 8 (64bits) or did you meant to align on 64bytes to better align on cache lines? > > - Please fit the whole file within 79 columns, there are a number of > lines that go beyond that. done > > - A Rid is a 32 bit unsigned, no need to cast to (unsigned long), just > declare the format string as %u (unsigned) I thought that int and unsigned are only guaranteed to have 16bit as minimal size, but I would be happy to hear that newer Posix standards have increased this to 32bits. > > - Not introduced with this patch, but can you remove most of the debug > stuff going to syslog ? > I do not think we want to spam syslog with this low level stuff, we must > not confuse debugging with logging like it happens in samba-land. > If you really want to retain that stuff, wrap it around a macro that is > enabled only with a debugging option passed to configure or make. Added a new patch which shows the messages only if -DDEBUG is set. bye, Sumit -------------- next part -------------- From c98b60d92c66718ea2eda8065c15130fc229bdcf Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Fri, 6 Jul 2012 12:24:01 +0200 Subject: [PATCH 36/37] Add unit tests for ipa-kdb To improve the performance of some functions used for group filtering we need a way to see if changes really help or not. This patch adds unit tests for some of these functions and measures the execution time. --- daemons/ipa-kdb/Makefile.am | 30 +++ daemons/ipa-kdb/ipa_kdb_mspac.c | 12 +- daemons/ipa-kdb/tests/common_check.c | 110 +++++++++ daemons/ipa-kdb/tests/common_check.h | 48 ++++ daemons/ipa-kdb/tests/ipa_kdb_tests.c | 440 ++++++++++++++++++++++++++++++++++ util/dlinklist.h | 132 ++++++++++ 6 Dateien ge?ndert, 768 Zeilen hinzugef?gt(+), 4 Zeilen entfernt(-) create mode 100644 daemons/ipa-kdb/tests/common_check.c create mode 100644 daemons/ipa-kdb/tests/common_check.h create mode 100644 daemons/ipa-kdb/tests/ipa_kdb_tests.c create mode 100644 util/dlinklist.h diff --git a/daemons/ipa-kdb/Makefile.am b/daemons/ipa-kdb/Makefile.am index 17c090418ec5a0e2a39d948dc385d509c5d05321..8cf719673a049d60111c67c3c99d807c952c34fc 100644 --- a/daemons/ipa-kdb/Makefile.am +++ b/daemons/ipa-kdb/Makefile.am @@ -50,6 +50,36 @@ ipadb_la_LIBADD = \ $(NDRPAC_LIBS) \ $(NULL) +if HAVE_CHECK +TESTS = ipa_kdb_tests +check_PROGRAMS = ipa_kdb_tests +endif + +ipa_kdb_tests_SOURCES = \ + tests/ipa_kdb_tests.c \ + tests/common_check.c \ + ipa_kdb.c \ + ipa_kdb_common.c \ + ipa_kdb_mkey.c \ + ipa_kdb_passwords.c \ + ipa_kdb_principals.c \ + ipa_kdb_pwdpolicy.c \ + ipa_kdb_mspac.c \ + ipa_kdb_delegation.c \ + ipa_kdb_audit_as.c \ + $(KRB5_UTIL_SRCS) \ + $(NULL) +ipa_kdb_tests_CFLAGS = $(CHECK_CFLAGS) +ipa_kdb_tests_LDADD = \ + $(CHECK_LIBS) \ + $(KRB5_LIBS) \ + $(LDAP_LIBS) \ + $(NDRPAC_LIBS) \ + -lnss3 \ + -lkdb5 \ + -lsss_idmap \ + $(NULL) + dist_noinst_DATA = ipa_kdb.exports EXTRA_DIST = \ diff --git a/daemons/ipa-kdb/ipa_kdb_mspac.c b/daemons/ipa-kdb/ipa_kdb_mspac.c index b5346fed1230d02a88c94ab913507112990a1651..e4c2a4d1b188b355605ada8f5f08fdb11d5adc02 100644 --- a/daemons/ipa-kdb/ipa_kdb_mspac.c +++ b/daemons/ipa-kdb/ipa_kdb_mspac.c @@ -659,9 +659,9 @@ static char *gen_sid_string(TALLOC_CTX *memctx, struct dom_sid *dom_sid, return str; } -static int get_group_sids(TALLOC_CTX *memctx, - struct PAC_LOGON_INFO_CTR *logon_info, - char ***_group_sids) +int get_group_sids(TALLOC_CTX *memctx, + struct PAC_LOGON_INFO_CTR *logon_info, + char ***_group_sids) { int ret; size_t c; @@ -669,6 +669,10 @@ static int get_group_sids(TALLOC_CTX *memctx, struct dom_sid *domain_sid = NULL; char **group_sids = NULL; + if (logon_info == NULL || logon_info->info == NULL) { + return EINVAL; + } + domain_sid = dom_sid_dup(memctx, logon_info->info->info3.base.domain_sid); if (domain_sid == NULL) { krb5_klog_syslog(LOG_ERR, "dom_sid_dup failed"); @@ -730,7 +734,7 @@ done: return ret; } -static int add_groups(TALLOC_CTX *memctx, +int add_groups(TALLOC_CTX *memctx, struct PAC_LOGON_INFO_CTR *logon_info, size_t ipa_group_sids_count, struct dom_sid2 *ipa_group_sids) diff --git a/daemons/ipa-kdb/tests/common_check.c b/daemons/ipa-kdb/tests/common_check.c new file mode 100644 index 0000000000000000000000000000000000000000..671c61655edeea635421fb6b7a3b304043dd8185 --- /dev/null +++ b/daemons/ipa-kdb/tests/common_check.c @@ -0,0 +1,110 @@ +/* + SSSD + + Common utilities for check-based tests using talloc. + + Authors: + Martin Nagy + + Copyright (C) Red Hat, Inc 2009 + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . +*/ + +#include +#include +#include "tests/common_check.h" +//#include "util/util.h" +#include "dlinklist.h" + +TALLOC_CTX *global_talloc_context = NULL; + + +struct size_snapshot { + struct size_snapshot *prev; + struct size_snapshot *next; + + TALLOC_CTX *ctx; + size_t bytes_allocated; +}; + +static struct size_snapshot *snapshot_stack; + +void +_check_leaks(TALLOC_CTX *ctx, size_t bytes, const char *location) +{ + size_t bytes_allocated; + + bytes_allocated = talloc_total_size(ctx); + if (bytes_allocated != bytes) { + fprintf(stderr, "Leak report for %s:\n", location); + talloc_report_full(ctx, stderr); + fail("%s: memory leaks detected, %d bytes still allocated", + location, bytes_allocated - bytes); + } +} + +void +check_leaks_push(TALLOC_CTX *ctx) +{ + struct size_snapshot *snapshot; + + snapshot = talloc(NULL, struct size_snapshot); + snapshot->ctx = ctx; + snapshot->bytes_allocated = talloc_total_size(ctx); + DLIST_ADD(snapshot_stack, snapshot); +} + +void +_check_leaks_pop(TALLOC_CTX *ctx, const char *location) +{ + struct size_snapshot *snapshot; + TALLOC_CTX *old_ctx; + size_t bytes_allocated; + + if (snapshot_stack == NULL) { + fail("%s: trying to pop an empty stack"); + } + + snapshot = snapshot_stack; + DLIST_REMOVE(snapshot_stack, snapshot); + + old_ctx = snapshot->ctx; + bytes_allocated = snapshot->bytes_allocated; + + fail_if(old_ctx != ctx, "Bad push/pop order"); + + talloc_free(snapshot); + snapshot = NULL; + _check_leaks(old_ctx, bytes_allocated, location); +} + +void +leak_check_setup(void) +{ + talloc_enable_null_tracking(); + global_talloc_context = talloc_new(NULL); + fail_unless(global_talloc_context != NULL, "talloc_new failed"); + check_leaks_push(global_talloc_context); +} + +void +leak_check_teardown(void) +{ + check_leaks_pop(global_talloc_context); + if (snapshot_stack != NULL) { + fail("Exiting with a non-empty stack"); + } + check_leaks(global_talloc_context, 0); +} diff --git a/daemons/ipa-kdb/tests/common_check.h b/daemons/ipa-kdb/tests/common_check.h new file mode 100644 index 0000000000000000000000000000000000000000..a9b306e449149a37e68d6ee6cbe58f75db8a1bbc --- /dev/null +++ b/daemons/ipa-kdb/tests/common_check.h @@ -0,0 +1,48 @@ +/* + SSSD + + Common utilities for check-based tests using talloc. + + Authors: + Martin Nagy + + Copyright (C) Red Hat, Inc 2009 + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . +*/ + +#ifndef __TESTS_COMMON_H__ +#define __TESTS_COMMON_H__ + +#include +#include "dlinklist.h" +//#include "providers/data_provider.h" +//#include "providers/ldap/sdap.h" + +extern TALLOC_CTX *global_talloc_context; + +#define check_leaks(ctx, bytes) _check_leaks((ctx), (bytes), __location__) +void _check_leaks(TALLOC_CTX *ctx, + size_t bytes, + const char *location); + +void check_leaks_push(TALLOC_CTX *ctx); + +#define check_leaks_pop(ctx) _check_leaks_pop((ctx), __location__) +void _check_leaks_pop(TALLOC_CTX *ctx, const char *location); + +void leak_check_setup(void); +void leak_check_teardown(void); + +#endif /* !__TESTS_COMMON_H__ */ diff --git a/daemons/ipa-kdb/tests/ipa_kdb_tests.c b/daemons/ipa-kdb/tests/ipa_kdb_tests.c new file mode 100644 index 0000000000000000000000000000000000000000..e8941ae7c50f2d6d66b08329a73f5b5e99f08eb5 --- /dev/null +++ b/daemons/ipa-kdb/tests/ipa_kdb_tests.c @@ -0,0 +1,440 @@ +/** BEGIN COPYRIGHT BLOCK + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + * + * Additional permission under GPLv3 section 7: + * + * In the following paragraph, "GPL" means the GNU General Public + * License, version 3 or any later version, and "Non-GPL Code" means + * code that is governed neither by the GPL nor a license + * compatible with the GPL. + * + * You may link the code of this Program with Non-GPL Code and convey + * linked combinations including the two, provided that such Non-GPL + * Code only links to the code of this Program through those well + * defined interfaces identified in the file named EXCEPTION found in + * the source code files (the "Approved Interfaces"). The files of + * Non-GPL Code may instantiate templates or use macros or inline + * functions from the Approved Interfaces without causing the resulting + * work to be covered by the GPL. Only the copyright holders of this + * Program may make changes or additions to the list of Approved + * Interfaces. + * + * Authors: + * Sumit Bose + * + * Copyright (C) 2012 Red Hat, Inc. + * All rights reserved. + * END COPYRIGHT BLOCK **/ + +#include +#include +#include +#include +#include +#include + +#include "gen_ndr/ndr_krb5pac.h" + +#include "ipa_kdb.h" +#include "tests/common_check.h" + +struct sss_idmap_ctx *global_idmap_ctx = NULL; + +#define DOM_SID "S-1-5-21-1234-5678-9012" +#define OTHER_DOM_SID "S-1-5-21-1111-2222-3333" +#define PRIMARY_GID 1234 +#define RID_BASE 10000 + +int krb5_klog_syslog(int l, const char *format, ...) +{ + va_list ap; + char *s = NULL; + int ret; + + va_start(ap, format); + + ret = vasprintf(&s, format, ap); + va_end(ap); + if (ret < 0) { + /* ENOMEM */ + return -1; + } + + fprintf(stderr, "%s\n", s); + free(s); + + return 0; +} + +int get_group_sids(TALLOC_CTX *memctx, + struct PAC_LOGON_INFO_CTR *logon_info, + char ***_group_sids); + +int add_groups(TALLOC_CTX *memctx, + struct PAC_LOGON_INFO_CTR *logon_info, + size_t ipa_group_sids_count, + struct dom_sid2 *ipa_group_sids); + +static void *idmap_talloc(size_t size, void *pvt) +{ + return talloc_size(pvt, size); +} + +static void idmap_free(void *ptr, void *pvt) +{ + talloc_free(ptr); +} + +static int make_dom_sid_array(TALLOC_CTX *mem_ctx, size_t count, + struct dom_sid **_sids) +{ + size_t c; + char *sid_str; + enum idmap_error_code err; + struct dom_sid *sids; + struct dom_sid *sid; + + sids = talloc_array(mem_ctx, struct dom_sid, count); + if (sids == NULL) { + return ENOMEM; + } + + for (c = 0; c < count; c++) { + sid_str = talloc_asprintf(mem_ctx, "%s-%lu", OTHER_DOM_SID, + (unsigned long) RID_BASE + c); + if (sid_str == NULL) { + return ENOMEM; + } + err = sss_idmap_sid_to_smb_sid(global_idmap_ctx, sid_str, &sid); + talloc_free(sid_str); + if (err != IDMAP_SUCCESS) { + return EINVAL; + } + memcpy(&sids[c], sid, sizeof(struct dom_sid)); + talloc_free(sid); + } + + *_sids = sids; + return 0; +} + +static int make_login_info(TALLOC_CTX *mem_ctx, size_t d_group_count, + size_t e_group_count, + struct PAC_LOGON_INFO_CTR **_logon_info) +{ + struct PAC_LOGON_INFO_CTR *logon_info; + enum idmap_error_code err; + size_t c; + char *sid_str; + struct dom_sid *sid; + + logon_info = talloc_zero(mem_ctx, struct PAC_LOGON_INFO_CTR); + if (logon_info == NULL) { + return ENOMEM; + } + + logon_info->info = talloc_zero(logon_info, struct PAC_LOGON_INFO); + if (logon_info->info == NULL) { + return ENOMEM; + } + + err = sss_idmap_sid_to_smb_sid(global_idmap_ctx, DOM_SID, &sid); + if (err != IDMAP_SUCCESS) { + return EINVAL; + } + logon_info->info->info3.base.domain_sid = talloc_steal(logon_info->info, + sid); + + logon_info->info->info3.base.primary_gid = PRIMARY_GID; + + if (d_group_count > 0) { + logon_info->info->info3.base.groups.count = d_group_count; + logon_info->info->info3.base.groups.rids = talloc_array(logon_info, + struct samr_RidWithAttribute, + d_group_count); + if (logon_info->info->info3.base.groups.rids == NULL) { + return ENOMEM; + } + + for (c = 0; c < d_group_count; c++) { + logon_info->info->info3.base.groups.rids[c].rid = RID_BASE + c; + logon_info->info->info3.base.groups.rids[c].attributes = 0; + } + } + + if (e_group_count > 0) { + logon_info->info->info3.sidcount = e_group_count; + logon_info->info->info3.sids = talloc_array(logon_info, + struct netr_SidAttr, + e_group_count); + if (logon_info->info->info3.sids == NULL) { + return ENOMEM; + } + + for (c = 0; c < e_group_count; c++) { + sid_str = talloc_asprintf(mem_ctx, "%s-%lu", OTHER_DOM_SID, + (unsigned long) RID_BASE + c); + if (sid_str == NULL) { + return ENOMEM; + } + err = sss_idmap_sid_to_smb_sid(global_idmap_ctx, sid_str, &sid); + talloc_free(sid_str); + if (err != IDMAP_SUCCESS) { + return EINVAL; + } + logon_info->info->info3.sids[c].sid = talloc_steal(logon_info->info, + sid); + logon_info->info->info3.sids[c].attributes = 0; + } + } + + *_logon_info = logon_info; + return 0; +} + +static int check_groups(size_t d_group_count, size_t e_group_count, + char **group_sids) +{ + int ret; + TALLOC_CTX *tmp_ctx; + char *tmp_str = NULL; + size_t c; + + tmp_ctx = talloc_new(NULL); + if (tmp_ctx == NULL) { + ret = ENOMEM; + goto done; + } + + tmp_str = talloc_asprintf(tmp_ctx, "%s-%d", DOM_SID, PRIMARY_GID); + if (tmp_str == NULL) { + ret = ENOMEM; + goto done; + } + + if (strcmp(group_sids[0], tmp_str) != 0) { + ret = EINVAL; + goto done; + } + + for (c = 0; c < d_group_count; c++) { + tmp_str = talloc_asprintf(tmp_ctx, "%s-%lu", DOM_SID, + (unsigned long) RID_BASE + c); + if (tmp_str == NULL) { + ret = ENOMEM; + goto done; + } + + if (strcmp(group_sids[1 + c], tmp_str) != 0) { + fprintf(stderr, "[%s][%s]\n", group_sids[1 + c], tmp_str); + ret = EINVAL; + goto done; + } + } + + for (c = 0; c < e_group_count; c++) { + tmp_str = talloc_asprintf(tmp_ctx, "%s-%lu", OTHER_DOM_SID, + (unsigned long) RID_BASE + c); + if (tmp_str == NULL) { + ret = ENOMEM; + goto done; + } + + if (strcmp(group_sids[1 + d_group_count + c], tmp_str) != 0) { + fprintf(stderr, "[%s][%s]\n", group_sids[1 + d_group_count + c], + tmp_str); + ret = EINVAL; + goto done; + } + } + + ret = 0; + +done: + talloc_free(tmp_ctx); + return ret; +} + +struct get_groups_test_params { + size_t dom_group_count; + size_t external_group_count; +}; + +START_TEST(test_get_groups) +{ + struct PAC_LOGON_INFO_CTR *logon_info = NULL; + char **group_sids = NULL; + int ret; + size_t c; + struct timeval start_time; + struct timeval end_time; + struct timeval diff_time; + struct get_groups_test_params params[] = { + {0, 0}, + {1000, 0}, + {10000, 0}, + {100000, 0}, + {1000000, 0}, + {0, 1000}, + {0, 10000}, + {0, 100000}, + {0, 1000000}, + {1000, 1000}, + {10000, 10000}, + {100000, 100000}, + {1000000, 1000000}, + {(size_t) -1, (size_t) -1} + }; + + for (c = 0; params[c].dom_group_count != (size_t) - 1; c++) { + ret = make_login_info(global_talloc_context, params[c].dom_group_count, + params[c].external_group_count, &logon_info); + fail_unless(ret == 0, "make_login_info failed."); + + ret = gettimeofday(&start_time, NULL); + fail_unless(ret == 0, "gettimeofday failed."); + + ret = get_group_sids(global_talloc_context, logon_info, &group_sids); + fail_unless(ret == 0, "get_group_sids failed [%d][%s].", + ret, strerror(ret)); + + ret = gettimeofday(&end_time, NULL); + fail_unless(ret == 0, "gettimeofday failed."); + + talloc_free(logon_info); + logon_info = NULL; + + fail_unless(check_groups(params[c].dom_group_count, + params[c].external_group_count, + group_sids) == 0, + "Unexpected SIDs returned."); + + talloc_free(group_sids); + group_sids = NULL; + + timersub(&end_time, &start_time, &diff_time); + fprintf(stderr, "get_groups: domain groups [%d] external group [%d] " \ + "duration [%ds %dus]\n", + params[c].dom_group_count, + params[c].external_group_count, + (int) diff_time.tv_sec, + (int) diff_time.tv_usec); + + } + +} +END_TEST + +struct add_groups_test_params { + size_t dom_group_count; +}; + +START_TEST(test_add_groups) +{ + struct PAC_LOGON_INFO_CTR *logon_info = NULL; + struct dom_sid *group_sids = NULL; + int ret; + size_t c; + struct timeval start_time; + struct timeval end_time; + struct timeval diff_time; + struct add_groups_test_params params[] = { + {0}, + {1000}, + {10000}, + {100000}, + {1000000}, + {2000000}, + {(size_t) -1} + }; + + for (c = 0; params[c].dom_group_count != (size_t) - 1; c++) { + ret = make_login_info(global_talloc_context, 10, 10, &logon_info); + fail_unless(ret == 0, "make_login_info failed."); + + ret = make_dom_sid_array(global_talloc_context, + params[c].dom_group_count, &group_sids); + fail_unless(ret == 0, "make_dom_sid_array failed."); + + ret = gettimeofday(&start_time, NULL); + fail_unless(ret == 0, "gettimeofday failed."); + + ret = add_groups(global_talloc_context, logon_info, + params[c].dom_group_count, group_sids); + fail_unless(ret == 0, "add_groups failed."); + + ret = gettimeofday(&end_time, NULL); + fail_unless(ret == 0, "gettimeofday failed."); + + talloc_free(group_sids); + group_sids = NULL; + talloc_free(logon_info); + logon_info = NULL; + + timersub(&end_time, &start_time, &diff_time); + fprintf(stderr, "add_groups: domain groups [%d] duration [%ds %dus]\n", + params[c].dom_group_count, + (int) diff_time.tv_sec, + (int) diff_time.tv_usec); + } + +} +END_TEST + +static void idmap_setup(void) +{ + enum idmap_error_code err; + + err = sss_idmap_init(idmap_talloc, global_talloc_context, idmap_free, + &global_idmap_ctx); + fail_unless(err == IDMAP_SUCCESS, "sss_idmap_init failed."); +} + +static void idmap_teardown(void) +{ + talloc_free(global_idmap_ctx); + global_idmap_ctx = NULL; +} + +Suite * ipa_kdb_suite(void) +{ + Suite *s = suite_create("IPA kdb"); + + TCase *tc_group = tcase_create("Group filtering"); + tcase_add_checked_fixture(tc_group, + leak_check_setup, + leak_check_teardown); + tcase_add_checked_fixture(tc_group, + idmap_setup, + idmap_teardown); + tcase_add_test(tc_group, test_get_groups); + tcase_add_test(tc_group, test_add_groups); + suite_add_tcase(s, tc_group); + + tcase_set_timeout(tc_group, 300); + return s; +} + +int main(void) +{ + int number_failed; + + Suite *s = ipa_kdb_suite (); + SRunner *sr = srunner_create (s); + srunner_run_all (sr, CK_VERBOSE); + number_failed = srunner_ntests_failed (sr); + srunner_free (sr); + + return (number_failed == 0) ? EXIT_SUCCESS : EXIT_FAILURE; +} diff --git a/util/dlinklist.h b/util/dlinklist.h new file mode 100644 index 0000000000000000000000000000000000000000..fa4b722559185ab2eb35c57cd84c9cb61a30fdb9 --- /dev/null +++ b/util/dlinklist.h @@ -0,0 +1,132 @@ +/* + Unix SMB/CIFS implementation. + some simple double linked list macros + Copyright (C) Andrew Tridgell 1998 + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . +*/ + +/* To use these macros you must have a structure containing a next and + prev pointer */ + +#ifndef _DLINKLIST_H +#define _DLINKLIST_H + + +/* hook into the front of the list */ +#define DLIST_ADD(list, p) \ +do { \ + if (!(list)) { \ + (list) = (p); \ + (p)->next = (p)->prev = NULL; \ + } else { \ + (list)->prev = (p); \ + (p)->next = (list); \ + (p)->prev = NULL; \ + (list) = (p); \ + }\ +} while (0) + +/* remove an element from a list - element doesn't have to be in list. */ +#define DLIST_REMOVE(list, p) \ +do { \ + if ((p) == (list)) { \ + (list) = (p)->next; \ + if (list) (list)->prev = NULL; \ + } else { \ + if ((p)->prev) (p)->prev->next = (p)->next; \ + if ((p)->next) (p)->next->prev = (p)->prev; \ + } \ + if ((p) != (list)) (p)->next = (p)->prev = NULL; \ +} while (0) + +/* promote an element to the top of the list */ +#define DLIST_PROMOTE(list, p) \ +do { \ + DLIST_REMOVE(list, p); \ + DLIST_ADD(list, p); \ +} while (0) + +/* hook into the end of the list - needs a tmp pointer */ +#define DLIST_ADD_END(list, p, type) \ +do { \ + if (!(list)) { \ + (list) = (p); \ + (p)->next = (p)->prev = NULL; \ + } else { \ + type tmp; \ + for (tmp = (list); tmp->next; tmp = tmp->next) ; \ + tmp->next = (p); \ + (p)->next = NULL; \ + (p)->prev = tmp; \ + } \ +} while (0) + +/* insert 'p' after the given element 'el' in a list. If el is NULL then + this is the same as a DLIST_ADD() */ +#define DLIST_ADD_AFTER(list, p, el) \ +do { \ + if (!(list) || !(el)) { \ + DLIST_ADD(list, p); \ + } else { \ + p->prev = el; \ + p->next = el->next; \ + el->next = p; \ + if (p->next) p->next->prev = p; \ + }\ +} while (0) + +/* demote an element to the end of the list, needs a tmp pointer */ +#define DLIST_DEMOTE(list, p, type) \ +do { \ + DLIST_REMOVE(list, p); \ + DLIST_ADD_END(list, p, type); \ +} while (0) + +/* concatenate two lists - putting all elements of the 2nd list at the + end of the first list */ +#define DLIST_CONCATENATE(list1, list2, type) \ +do { \ + if (!(list1)) { \ + (list1) = (list2); \ + } else { \ + type tmp; \ + for (tmp = (list1); tmp->next; tmp = tmp->next) ; \ + tmp->next = (list2); \ + if (list2) { \ + (list2)->prev = tmp; \ + } \ + } \ +} while (0) + +/* insert all elements from list2 after the given element 'el' in the + * first list */ +#define DLIST_ADD_LIST_AFTER(list1, el, list2, type) \ +do { \ + if (!(list1) || !(el) || !(list2)) { \ + DLIST_CONCATENATE(list1, list2, type); \ + } else { \ + type tmp; \ + for (tmp = (list2); tmp->next; tmp = tmp->next) ; \ + (list2)->prev = (el); \ + tmp->next = (el)->next; \ + (el)->next = (list2); \ + if (tmp->next != NULL) tmp->next->prev = tmp; \ + } \ +} while (0); + +#define DLIST_FOR_EACH(p, list) \ + for ((p) = (list); (p) != NULL; (p) = (p)->next) + +#endif /* _DLINKLIST_H */ -- 1.7.11.4 -------------- next part -------------- From b0ff2e6260747a04bf68a9b9175ab2a5cdd403b5 Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Fri, 6 Jul 2012 21:40:35 +0200 Subject: [PATCH 37/37] Improve performance of get_group_sids() This patch tries to improve the performance of get_group_sids() by reducing the number of memory allocations. --- daemons/ipa-kdb/ipa_kdb_mspac.c | 112 ++++++++++++++-------------------------- 1 Datei ge?ndert, 38 Zeilen hinzugef?gt(+), 74 Zeilen entfernt(-) diff --git a/daemons/ipa-kdb/ipa_kdb_mspac.c b/daemons/ipa-kdb/ipa_kdb_mspac.c index e4c2a4d1b188b355605ada8f5f08fdb11d5adc02..a8f56f3a6ff376b74f69a01e64a2d3de7c1a80d2 100644 --- a/daemons/ipa-kdb/ipa_kdb_mspac.c +++ b/daemons/ipa-kdb/ipa_kdb_mspac.c @@ -86,6 +86,7 @@ static char *memberof_pac_attrs[] = { #define SID_ID_AUTHS 6 #define SID_SUB_AUTHS 15 #define MAX(a,b) (((a)>(b))?(a):(b)) +#define ALIGN_BASE 8 static int string_to_sid(char *str, struct dom_sid *sid) { @@ -185,43 +186,6 @@ static char *dom_sid_string(TALLOC_CTX *memctx, const struct dom_sid *dom_sid) return buf; } -static struct dom_sid *dom_sid_dup(TALLOC_CTX *memctx, - const struct dom_sid *dom_sid) -{ - struct dom_sid *new_sid; - size_t c; - - if (dom_sid == NULL) { - return NULL; - } - - new_sid = talloc(memctx, struct dom_sid); - if (new_sid == NULL) { - return NULL; - } - - new_sid->sid_rev_num = dom_sid->sid_rev_num; - for (c = 0; c < SID_ID_AUTHS; c++) { - new_sid->id_auth[c] = dom_sid->id_auth[c]; - } - new_sid->num_auths = dom_sid->num_auths; - for (c = 0; c < SID_SUB_AUTHS; c++) { - new_sid->sub_auths[c] = dom_sid->sub_auths[c]; - } - - return new_sid; -} - -static int sid_append_rid(struct dom_sid *sid, uint32_t rid) -{ - if (sid->num_auths >= SID_SUB_AUTHS) { - return EINVAL; - } - - sid->sub_auths[sid->num_auths++] = rid; - return 0; -} - /** * @brief Takes a user sid and removes the rid. * The sid is changed by this function, @@ -636,29 +600,6 @@ static bool is_cross_realm_krbtgt(krb5_const_principal princ) return true; } -static char *gen_sid_string(TALLOC_CTX *memctx, struct dom_sid *dom_sid, - uint32_t rid) -{ - char *str = NULL; - int ret; - - ret = sid_append_rid(dom_sid, rid); - if (ret != 0) { - krb5_klog_syslog(LOG_ERR, "sid_append_rid failed"); - return NULL; - } - - str = dom_sid_string(memctx, dom_sid); - ret = sid_split_rid(dom_sid, NULL); - if (ret != 0) { - krb5_klog_syslog(LOG_ERR, "sid_split_rid failed"); - talloc_free(str); - return NULL; - } - - return str; -} - int get_group_sids(TALLOC_CTX *memctx, struct PAC_LOGON_INFO_CTR *logon_info, char ***_group_sids) @@ -666,19 +607,28 @@ int get_group_sids(TALLOC_CTX *memctx, int ret; size_t c; size_t p = 0; - struct dom_sid *domain_sid = NULL; + char *dom_sid_str = NULL; char **group_sids = NULL; + char *group_sids_buf = NULL; + size_t dom_sid_str_len; + size_t sid_len; + char *idx; + if (logon_info == NULL || logon_info->info == NULL) { return EINVAL; } - domain_sid = dom_sid_dup(memctx, logon_info->info->info3.base.domain_sid); - if (domain_sid == NULL) { - krb5_klog_syslog(LOG_ERR, "dom_sid_dup failed"); + dom_sid_str = dom_sid_string(memctx, + logon_info->info->info3.base.domain_sid); + if (dom_sid_str == NULL) { + krb5_klog_syslog(LOG_ERR, "dom_sid_string failed."); ret = ENOMEM; goto done; } + dom_sid_str_len = strlen(dom_sid_str); + sid_len = dom_sid_str_len + 12; + sid_len = sid_len + ( ALIGN_BASE - ( sid_len % ALIGN_BASE)); group_sids = talloc_array(memctx, char *, 2 + @@ -690,27 +640,41 @@ int get_group_sids(TALLOC_CTX *memctx, goto done; } - group_sids[p] = gen_sid_string(memctx, domain_sid, - logon_info->info->info3.base.primary_gid); - if (group_sids[p] == NULL) { - krb5_klog_syslog(LOG_ERR, "gen_sid_string failed"); + group_sids_buf = talloc_zero_size(group_sids, + sid_len * (1 + logon_info->info->info3.base.groups.count)); + if (group_sids_buf == NULL) { + krb5_klog_syslog(LOG_ERR, "talloc_zero failed"); + ret = ENOMEM; + goto done; + } + + idx = group_sids_buf + (p * sid_len); + memcpy(idx, dom_sid_str, dom_sid_str_len); + ret = snprintf(idx + dom_sid_str_len, 12, "-%lu", + (unsigned long) logon_info->info->info3.base.primary_gid); + if (ret < 0 || ret >= 12) { + krb5_klog_syslog(LOG_ERR, "snprintf failed"); ret = EINVAL; goto done; } + group_sids[p] = idx; p++; for (c = 0; c < logon_info->info->info3.base.groups.count; c++) { - group_sids[p] = gen_sid_string(memctx, domain_sid, - logon_info->info->info3.base.groups.rids[c].rid); - if (group_sids[p] == NULL) { - krb5_klog_syslog(LOG_ERR, "gen_sid_string 2 failed"); + idx = group_sids_buf + (p * sid_len); + memcpy(idx, dom_sid_str, dom_sid_str_len); + ret = snprintf(idx + dom_sid_str_len, 12, "-%lu", + (unsigned long) logon_info->info->info3.base.groups.rids[c].rid); + if (ret < 0 || ret >= 12) { + krb5_klog_syslog(LOG_ERR, "snprintf failed"); ret = EINVAL; goto done; } + group_sids[p] = idx; p++; } for (c = 0; c < logon_info->info->info3.sidcount; c++) { - group_sids[p] = dom_sid_string(memctx, + group_sids[p] = dom_sid_string(group_sids, logon_info->info->info3.sids[c].sid); if (group_sids[p] == NULL) { krb5_klog_syslog(LOG_ERR, "dom_sid_string failed"); @@ -726,7 +690,7 @@ int get_group_sids(TALLOC_CTX *memctx, ret = 0; done: - talloc_free(domain_sid); + talloc_free(dom_sid_str); if (ret != 0) { talloc_free(group_sids); } -- 1.7.11.4 -------------- next part -------------- From 87ce1bc1d0d7508f3ed79da58972d0e1cc8b1f11 Mon Sep 17 00:00:00 2001 From: Sumit Bose Date: Thu, 27 Sep 2012 15:33:55 +0200 Subject: [PATCH] ipadb: Show debug messages only in a debug build --- daemons/ipa-kdb/ipa_kdb_mspac.c | 56 ++++++++++++++++++++++++++--------------- 1 Datei ge?ndert, 36 Zeilen hinzugef?gt(+), 20 Zeilen entfernt(-) diff --git a/daemons/ipa-kdb/ipa_kdb_mspac.c b/daemons/ipa-kdb/ipa_kdb_mspac.c index a8f56f3a6ff376b74f69a01e64a2d3de7c1a80d2..b3cef97db1bfe120d00b3bbfd30ed392dbb7b385 100644 --- a/daemons/ipa-kdb/ipa_kdb_mspac.c +++ b/daemons/ipa-kdb/ipa_kdb_mspac.c @@ -42,9 +42,25 @@ struct ipadb_mspac { struct ipadb_adtrusts *trusts; }; - +#ifdef DEBUG int krb5_klog_syslog(int, const char *, ...); +void debug_fn(const char *format, ...) +{ + va_list ap; + + va_start(ap, format); + + krb5_klog_syslog(LOG_ERR, format, ap); + + va_end(ap); +} + +#define KDB_DEBUG(body) debug_fn body +#else +#define KDB_DEBUG(body) +#endif + static char *user_pac_attrs[] = { "objectClass", "uid", @@ -622,7 +638,7 @@ int get_group_sids(TALLOC_CTX *memctx, dom_sid_str = dom_sid_string(memctx, logon_info->info->info3.base.domain_sid); if (dom_sid_str == NULL) { - krb5_klog_syslog(LOG_ERR, "dom_sid_string failed."); + KDB_DEBUG(("dom_sid_string failed.")); ret = ENOMEM; goto done; } @@ -635,7 +651,7 @@ int get_group_sids(TALLOC_CTX *memctx, logon_info->info->info3.base.groups.count + logon_info->info->info3.sidcount); if (group_sids == NULL) { - krb5_klog_syslog(LOG_ERR, "talloc_array failed"); + KDB_DEBUG(("talloc_array failed")); ret = ENOMEM; goto done; } @@ -643,7 +659,7 @@ int get_group_sids(TALLOC_CTX *memctx, group_sids_buf = talloc_zero_size(group_sids, sid_len * (1 + logon_info->info->info3.base.groups.count)); if (group_sids_buf == NULL) { - krb5_klog_syslog(LOG_ERR, "talloc_zero failed"); + KDB_DEBUG(("talloc_zero failed")); ret = ENOMEM; goto done; } @@ -653,7 +669,7 @@ int get_group_sids(TALLOC_CTX *memctx, ret = snprintf(idx + dom_sid_str_len, 12, "-%lu", (unsigned long) logon_info->info->info3.base.primary_gid); if (ret < 0 || ret >= 12) { - krb5_klog_syslog(LOG_ERR, "snprintf failed"); + KDB_DEBUG(("snprintf failed")); ret = EINVAL; goto done; } @@ -666,7 +682,7 @@ int get_group_sids(TALLOC_CTX *memctx, ret = snprintf(idx + dom_sid_str_len, 12, "-%lu", (unsigned long) logon_info->info->info3.base.groups.rids[c].rid); if (ret < 0 || ret >= 12) { - krb5_klog_syslog(LOG_ERR, "snprintf failed"); + KDB_DEBUG(("snprintf failed")); ret = EINVAL; goto done; } @@ -677,7 +693,7 @@ int get_group_sids(TALLOC_CTX *memctx, group_sids[p] = dom_sid_string(group_sids, logon_info->info->info3.sids[c].sid); if (group_sids[p] == NULL) { - krb5_klog_syslog(LOG_ERR, "dom_sid_string failed"); + KDB_DEBUG(("dom_sid_string failed")); ret = EINVAL; goto done; } @@ -763,7 +779,7 @@ static int map_groups(TALLOC_CTX *memctx, krb5_context kcontext, basedn = talloc_asprintf(memctx, "cn=groups,cn=accounts,%s", ipactx->base); if (basedn == NULL) { - krb5_klog_syslog(LOG_ERR, "talloc_asprintf failed."); + KDB_DEBUG(("talloc_asprintf failed.")); kerr = ENOMEM; goto done; } @@ -773,7 +789,7 @@ static int map_groups(TALLOC_CTX *memctx, krb5_context kcontext, filter = talloc_asprintf(memctx, "(&(objectclass=ipaExternalGroup)(ipaExternalMember=%s))", group_sids[c]); if (filter == NULL) { - krb5_klog_syslog(LOG_ERR, "talloc_asprintf failed."); + KDB_DEBUG(("talloc_asprintf failed.")); kerr = ENOMEM; goto done; } @@ -782,7 +798,7 @@ static int map_groups(TALLOC_CTX *memctx, krb5_context kcontext, entry_attrs, deref_search_attrs, memberof_pac_attrs, &results); if (kerr != 0) { - krb5_klog_syslog(LOG_ERR, "ipadb_deref_search failed."); + KDB_DEBUG(("ipadb_deref_search failed.")); goto done; } @@ -799,7 +815,7 @@ static int map_groups(TALLOC_CTX *memctx, krb5_context kcontext, break; case 0: if (deref_results == NULL) { - krb5_klog_syslog(LOG_ERR, "No results."); + KDB_DEBUG(("No results.")); break; } @@ -809,7 +825,7 @@ static int map_groups(TALLOC_CTX *memctx, krb5_context kcontext, sids = talloc_realloc(memctx, sids, struct dom_sid, count); if (sids == NULL) { - krb5_klog_syslog(LOG_ERR, "talloc_realloc failed."); + KDB_DEBUG(("talloc_realloc failed.")); kerr = ENOMEM; goto done; } @@ -838,9 +854,9 @@ static int map_groups(TALLOC_CTX *memctx, krb5_context kcontext, if (gid != 0 && sid.sid_rev_num != 0) { /* TODO: check if gid maps to sid */ if (sid_index >= count) { - krb5_klog_syslog(LOG_ERR, "Index larger than " + KDB_DEBUG(("Index larger than " "array, this shoould " - "never happen."); + "never happen.")); kerr = EFAULT; goto done; } @@ -911,7 +927,7 @@ static krb5_error_code add_local_groups(krb5_context context, ret = add_groups(memctx, info, ipa_group_sids_count, ipa_group_sids); if (ret != 0) { - krb5_klog_syslog(LOG_ERR, "add_groups failed"); + KDB_DEBUG(("add_groups failed")); return KRB5_KDB_INTERNAL_ERROR; } @@ -996,11 +1012,11 @@ static krb5_error_code filter_logon_info(krb5_context context, /* check netbios/flat name */ if (strcasecmp(info->info->info3.base.logon_domain.string, domain->flat_name) != 0) { - krb5_klog_syslog(LOG_ERR, "PAC Info mismatch: domain = %s, " + KDB_DEBUG(("PAC Info mismatch: domain = %s, " "expected flat name = %s, " "found logon name = %s", domain->domain_name, domain->flat_name, - info->info->info3.base.logon_domain.string); + info->info->info3.base.logon_domain.string)); return EINVAL; } @@ -1011,11 +1027,11 @@ static krb5_error_code filter_logon_info(krb5_context context, } if (strcmp(domsid, domain->domain_sid) != 0) { - krb5_klog_syslog(LOG_ERR, "PAC Info mismatch: domain = %s, " + KDB_DEBUG(("PAC Info mismatch: domain = %s, " "expected domain SID = %s, " "found domain SID = %s", domain->domain_name, domain->domain_sid, - domsid); + domsid)); talloc_free(domsid); return EINVAL; } @@ -1500,7 +1516,7 @@ krb5_error_code ipadb_mspac_get_trusted_domains(struct ipadb_context *ipactx) done: if (ret != 0) { - krb5_klog_syslog(LOG_ERR, "Failed to read list of trusted domains"); + KDB_DEBUG(("Failed to read list of trusted domains")); } free(base); return ret; -- 1.7.11.4