Discussion:
Services draft goals: Request for Comments
Gabriel Wicke
2014-06-05 17:21:29 UTC
Permalink
Hi all,

Matt & I have published our draft goals for the nascent Services team [1] at

https://www.mediawiki.org/wiki/Services/Roadmap

Your feedback would be much appreciated.

Thanks,

Gabriel

[1]: https://www.mediawiki.org/wiki/Services
Faidon Liambotis
2014-06-08 03:30:27 UTC
Permalink
Post by Gabriel Wicke
Matt & I have published our draft goals for the nascent Services team [1] at
https://www.mediawiki.org/wiki/Services/Roadmap
Your feedback would be much appreciated.
- These goals are vast and in very exciting in general. Clearly the team
has a vision and a lot of thought has been put into compiling this big
list of tasks. Kudos to both of you for writing this up and sharing it
with us, this is incredibly useful!

- At the same time, some of these goals are /too/ specific and do not
leave much up to interpretation and deliberation.

For some of these, we haven't agreed if they're towards the direction
that we're going to take (minor example: "leverage packages as much as
possible for deployment, DRY") and the goals process is not really the
proper forum to make those decisions.

Making them a bit more abstract ("work on improving our production
deployment strategy") would help keep our options open and avoid
misunderstandings when the time comes.

- On a more specific note, I'm still personally in doubt on whether the
MWCore team has signed off on a) the storage service plans b) static
Parsoid HTML5 for all page views plan, especially in the detail
articulated in your goals. As this is a big part of your plan for next
year, I think it's worth clarifying.

- This is *a lot* of work. Are you sure you can pull this off?

As more of these core services get deployed, you're going to incur
some of the costs supporting those deployments. This is essentially
the "ops problem" of scheduling work; both us and MWCore know well how
this can throw off our plans and schedules and we tend to account for
it. Have you considered this?

Moreover, my impression of the team's charter is that it will also
have a supportive role to other teams that want to write or use
services. I see you've accounted for some of that (e.g. mobile) but I
think you should probably expect more of it as SOA catches up within
the foundation.

Finally, I think you should expect some (healthy, I hope!) amount of
debate and consideration for certain things you intend to do,
something that can also be quite time-consuming, obviously :)

(what are the week numbers in parenthesis supposed to be, btw? Is this
FTE? If so, e.g. Q2 accounts for 18 weeks for Gabriel, probably a bit
too much for a single quarter ;))

- Several of the things listed there will require hardware, possibly a
non-trivial amount of it. Could you make a very rough list of your
expectations so that we can plan for it, or is it too early to tell?

Regards,
Faidon
Gabriel Wicke
2014-06-09 01:05:58 UTC
Permalink
Hi Faidon,

thank you for your input!
Post by Faidon Liambotis
Post by Gabriel Wicke
Matt & I have published our draft goals for the nascent Services team [1] at
https://www.mediawiki.org/wiki/Services/Roadmap
- At the same time, some of these goals are /too/ specific and do not
leave much up to interpretation and deliberation.
I hope that they are specific enough to invite critical input and identify
points of contention. These plans are also not immutable, and I'm happy to
make them less specific in areas where there's more need for discussion.
Post by Faidon Liambotis
- This is *a lot* of work. Are you sure you can pull this off?
I'm cautiously optimistic. Q1 and to some degree Q2 will be tight, but it's
important for other teams to get the basic functionality done early. We have
prototypes for parts of this, so have an idea of what we are getting
ourselves into. There are also some optional items in Q3 / Q4 that could be
canceled or deferred if earlier tasks take longer than expected.
Post by Faidon Liambotis
Moreover, my impression of the team's charter is that it will also
have a supportive role to other teams that want to write or use
services. I see you've accounted for some of that (e.g. mobile) but I
think you should probably expect more of it as SOA catches up within
the foundation.
The supportive role is indeed intended. Initially much of this support is
going to be in the form of building general infrastructure (storage service,
REST API, caching), but over time we'll spend more time on helping other
teams. Some of this will be supporting specific use cases, but we'll also
help other teams develop their own specialized services and API end points.
This experience will help us refine the general infrastructure, and might
also surface repeating problems that could be solved generically. Overall I
think that we should have enough capacity in Q3 & Q4 to work on this.
Post by Faidon Liambotis
Finally, I think you should expect some (healthy, I hope!) amount of
debate and consideration for certain things you intend to do,
something that can also be quite time-consuming, obviously :)
I'd be very disappointed if this was the end of all technical debate. ;)

But more seriously, I think a good amount of time is going to be needed for
working with the community. This is the reason why we didn't set a deadline
for using Parsoid HTML5 for page views. Our goal is to get the technical
preliminaries in place, and then test, refine and discuss things based on
the real thing. Much of this is probably going to be handled by Parsoid and
the Community Engagement team, but we'll still need to be prepared to make
technical adjustments as needed.
Post by Faidon Liambotis
(what are the week numbers in parenthesis supposed to be, btw? Is this
FTE? If so, e.g. Q2 accounts for 18 weeks for Gabriel, probably a bit
too much for a single quarter ;))
Indeed ;) These are (very rough) estimates of how long it might take one of
us to tackle individual tasks. So yes, FTEs with an element of planning poker.
Post by Faidon Liambotis
- Several of the things listed there will require hardware, possibly a
non-trivial amount of it. Could you make a very rough list of your
expectations so that we can plan for it, or is it too early to tell?
I don't expect major hardware needs for the storage service or REST API
before Q2. Until then we should be able to make do with little more than the
misc servers we've been using for testing so far. The smaller services
(mathoid, citations etc) will need two shared boxes for redundancy. Lets do
some more detailed planning on this soon.

Gabriel
Nuria Ruiz
2014-06-12 16:53:16 UTC
Permalink
Hello,

I came kind of late to this discussion so you guys might have already
talked about this, please ignore if that is the case.
I see that in the goals page there is a bunch of work regarding current
projects but there is little mentioning of the 'background work' needed to
deploy services. It seems that the work of building the infrastructure
should be called out explicitly.

For example:

1. Transport: have we chosen what is it going to be our transport protocol?
as in json-rpc, protocol buffers...

2. Authentication: are services for internal consumption only (and thus
inside our firewall) or do they provide an authentication scheme?

3. Logging: How would you follow requests across services, or at least,
follow the jump from client to server?

4.Versioning

5.Testing (version aware)

6.Client Configs, how does the client know where services are?

7.Code generation from service api descriptor
Post by Gabriel Wicke
Hi all,
Matt & I have published our draft goals for the nascent Services team [1] at
https://www.mediawiki.org/wiki/Services/Roadmap
Your feedback would be much appreciated.
Thanks,
Gabriel
[1]: https://www.mediawiki.org/wiki/Services
_______________________________________________
Ops mailing list
https://lists.wikimedia.org/mailman/listinfo/ops
Gabriel Wicke
2014-06-12 20:28:28 UTC
Permalink
Hi Nuria!
Post by Nuria Ruiz
Hello,
I came kind of late to this discussion so you guys might have already
talked about this, please ignore if that is the case.
Much of this has been discussed before, but I'm happy to give you pointers.
Post by Nuria Ruiz
I see that in the goals page there is a bunch of work regarding current
projects but there is little mentioning of the 'background work' needed to
deploy services. It seems that the work of building the infrastructure
should be called out explicitly.
I think some of that is already called out. We see it very much as one of
our responsibilities to improve practices across services. We also already
have some experience with production services like Parsoid, so we are not
starting from scratch.
Post by Nuria Ruiz
1. Transport: have we chosen what is it going to be our transport protocol?
as in json-rpc, protocol buffers...
We are aiming for more REST and less RPC. We'll start with JSON and HTML
resources, but could consider other representations later.
Post by Nuria Ruiz
2. Authentication: are services for internal consumption only (and thus
inside our firewall) or do they provide an authentication scheme?
As stated in the goals, we'll start the first phase with simple internal
authentication, but invest time in a proper solution later on in the fiscal,
in collaboration with platform. See
https://www.mediawiki.org/wiki/Requests_for_comment/SOA_Authentication.
Post by Nuria Ruiz
3. Logging: How would you follow requests across services, or at least,
follow the jump from client to server?
Matt just set up a GELF sink feeding into logstash. Tracking requests across
the infrastructure should be possible by including a unique request id in
the structured log information.
Post by Nuria Ruiz
4.Versioning
5.Testing (version aware)
https://www.mediawiki.org/wiki/Requests_for_comment/Storage_service#API_versioning

https://www.mediawiki.org/wiki/Services/Roadmap#Drive_automated_service_testing

Re versioning: I'm leaning towards testing version transformations
separately from the main functionality, as it reduces the number of cases we
need to cover.
Post by Nuria Ruiz
6.Client Configs, how does the client know where services are?
Are you more interested in discovery of internal services or external API
end points?
Post by Nuria Ruiz
7.Code generation from service api descriptor
Are you interested in clients or mocks?

Gabriel
Nuria Ruiz
2014-06-13 09:09:05 UTC
Permalink
I will do my best to try to take a look at the docs you sent and add
comments where it pertains.
Post by Gabriel Wicke
We are aiming for more REST and less RPC. We'll start with JSON and HTML
resources, but could consider other representations later.

I see, in case you did not see it the heroku guys jus released a very
useful set of guidelines for rest apis. The most interesting bit is
pagination, I think:
https://github.com/interagent/http-api-design
Post by Gabriel Wicke
Are you more interested in discovery of internal services or external API end
points?
Not discovery but rather configuration, so I can test serviceX locally in
vagrant and also deploy the client in prod and client has access to the
production configuration. In our case probably we resolve this with puppet
as it seems to be the mother of all things at wikimedia.
Post by Gabriel Wicke
Post by Nuria Ruiz
7.Code generation from service api descriptor
Client generation that can be used for both, see for example:
https://github.com/interagent/heroics
Post by Gabriel Wicke
Hi Nuria!
Post by Nuria Ruiz
Hello,
I came kind of late to this discussion so you guys might have already
talked about this, please ignore if that is the case.
Much of this has been discussed before, but I'm happy to give you pointers.
Post by Nuria Ruiz
I see that in the goals page there is a bunch of work regarding current
projects but there is little mentioning of the 'background work' needed
to
Post by Nuria Ruiz
deploy services. It seems that the work of building the infrastructure
should be called out explicitly.
I think some of that is already called out. We see it very much as one of
our responsibilities to improve practices across services. We also already
have some experience with production services like Parsoid, so we are not
starting from scratch.
Post by Nuria Ruiz
1. Transport: have we chosen what is it going to be our transport
protocol?
Post by Nuria Ruiz
as in json-rpc, protocol buffers...
We are aiming for more REST and less RPC. We'll start with JSON and HTML
resources, but could consider other representations later.
Post by Nuria Ruiz
2. Authentication: are services for internal consumption only (and thus
inside our firewall) or do they provide an authentication scheme?
As stated in the goals, we'll start the first phase with simple internal
authentication, but invest time in a proper solution later on in the fiscal,
in collaboration with platform. See
https://www.mediawiki.org/wiki/Requests_for_comment/SOA_Authentication.
Post by Nuria Ruiz
3. Logging: How would you follow requests across services, or at least,
follow the jump from client to server?
Matt just set up a GELF sink feeding into logstash. Tracking requests across
the infrastructure should be possible by including a unique request id in
the structured log information.
Post by Nuria Ruiz
4.Versioning
5.Testing (version aware)
https://www.mediawiki.org/wiki/Requests_for_comment/Storage_service#API_versioning
https://www.mediawiki.org/wiki/Services/Roadmap#Drive_automated_service_testing
Re versioning: I'm leaning towards testing version transformations
separately from the main functionality, as it reduces the number of cases we
need to cover.
Post by Nuria Ruiz
6.Client Configs, how does the client know where services are?
Are you more interested in discovery of internal services or external API
end points?
Post by Nuria Ruiz
7.Code generation from service api descriptor
Are you interested in clients or mocks?
Gabriel
Loading...